Codesmith’s Commitment to CIRR & Transparent Reporting: Frequently Asked Questions

Alumni speak of having joined Codesmith for many reasons (academic rigor, the tight-knit community, global network of engineers, to become founders, investors & CTOs) but many of these same alums would also likely tell you that transparent outcomes reporting via the Council on Integrity in Results Reporting (CIRR) was a major factor in their decision. 

And while it’s difficult to reduce an engineer’s career status to salary or “days to get hired,” CIRR reports offer a set of crucial data that allow prospective applicants to look ahead to their lives after graduation.

With that in mind, CIRR was established in 2016 as a 501(c)(6) non-profit to set shared standards on reporting student outcomes, focused entirely on reporting accurate and transparent outcomes (i.e., placement, salary within a fixed time of 180 days from graduation, etc.) that are verified by an AICPA ‘peer reviewed’ audit firm.

The standards are extremely tough (full details below and on CIRR’s site), which can be demanding and time-consuming for participating programs (and thus over the years the number of programs reporting to CIRR has decreased). But given the importance of the information, we’d always prefer that they remain stringent and uncompromising.

In addition, CIRR is strict with how graduates are classified. For example, if a student decides after the program not to engage in a job search, they will count against the ‘percentage employed in field’ statistic. And while the typical standard for finding employment following traditional higher ed is a year, CIRR has chosen 180 days as a very rigid cut off. 

It’s tempting for a program to develop their own, more flexible standards in order to adjust for edge cases that can lower outcome statistics (and many have), but we built Codesmith to be a space focused on engineering rigor and a tight-knit, supportive community, and part of that includes holding ourselves to these same high standards. 

FREQUENTLY ASKED QUESTIONS

Who is included in the report, and what kind of students are not included in the reports? 

Every single student that has attended Codesmith is included in CIRR reports (cohort sizes have ranged from 13 to 37 since we started reporting). If a student indicates that they do not plan to search for a job or are not legally authorized to work in the country where they are living after graduation, they indicate that to the team prior to the program start date, and that percentage is included in the report. The default assumption if not indicated is that graduates do plan to seek employment within 180 days of graduation. 

Who designed CIRR? 

CIRR was founded by Skills Fund in 2016 (now Ascent) - Codesmith was permitted to join in 2019 (once it had 2+ years of data) and joined the board in May 2022. The latest standards were established in 2020.

How does Codesmith collect graduate’s outcomes for CIRR? 

The Outcomes team at Codesmith prides itself on working hard to continually stay in touch with graduates of the program while supporting them throughout their job search. Most graduates will proactively let us know once they have signed an offer letter. When a graduate lets the team know an offer has been signed, they are sent a form to fill out that confirms their job information. This form is not affiliated with CIRR, but is used to build out the report later submitted to the auditing team. 

Although most graduates report their outcomes proactively, there are always a handful of graduates that have not reported new employment to us. Typically, some of those graduates are continuing to job search, some have gone on to work in different fields, a small percentage have had family/life emergencies that have required them to step away, and typically a number of graduates have accepted full-time employment, but did not report their offers to us. As such, the team has a process to reach out to all alumni and uses this information to determine the final category as outlined within CIRR standards using Tracking Job Outcomes. 

Codesmith recently reported our outcomes H1 2022  (for graduates that completed the program between January and June of 2022, and then completed a job search from July to December of 2022). During the data gathering process, we began by emailing all remaining graduates that had not informed us of an accepted offer. After the email had gone out, program coordinators that worked with the graduates reached out via text message to check in on their progress. As a final step, a senior hiring program team member does a final reach out to ensure we’ve done our due diligence to touch base multiple times with every single graduate of the program. 

While the team is touching base with graduates, it is also worth noting that these conversations are part of ongoing professional development strategy check-ins with alumni and additional one-on-one hiring support sessions. Codesmith hiring support is for life, and the team takes that commitment very seriously. 

During the 2022 H1 reporting period with an average of 80.1% in CIRR classification ‘Employed in field’, individual cohorts ranged from 65% to 95% - within some expected statistical variance. Much of the variation is in median days to offer and so the difference normalizes beyond the 180 day limit of CIRR. We’ll be publishing full 365 day report audited to the CIRR standard annually.

How many cohorts are included in each CIRR report? 

For the full-time remote program there are 9-12 cohorts graduating in a 6 month period. For part-time remote and NYC onsite 3-4 graduate in a 6 month period. Within the H1-2022 reports, cohort sizes varied from 17-36 graduates.

Prior to H1-2022, CIRR results were reported by campus - (West Coast, Central, East Coast and Part-Time Remote). For the full-time campuses (West Coast, Central and East Coast), CIRR reports included 3 to 4 cohorts (with cohorts starting every seven weeks, there are 3 or 4 cohorts graduating within a 6 month period). For the Part-Time program, CIRR reports included 1 to 2 cohorts (with cohorts starting every 12 weeks, there are 1 to 2 cohorts graduating within a 6 month period). The total number of students included in each report depends on how many students were in each cohort during that reporting period. Cohorts are typically around 30-36 students, so the full-time reports will usually range from approximately 102 - 144 students, while part-time reports will usually range from approximately 34 - 72 students. Note that in earlier reports (2021 and below) some cohort sizes were smaller, ranging from 13 to 37 students. 

Why do older reports have different program names?

To match the structure of full-time remote, part-time remote and NYC onsite, where the remote programs are not location specific (and people attend from all over the US and world), we produce one report for full-time remote, one for part-time remote, & will procure one for NYC onsite when the first 6 months of data is ready.

Previously CIRR reports were titled according to their legacy location: 

• LA or West Coast Remote Immersive

• NY or East Coast Remote Immersive

• Remote or Central Remote Immersive

• Global Part-Time or Part-Time Remote Immersive

What happens when Codesmith cannot get in touch with a graduate? 

While we contact every single graduate multiple times, there may be a small handful we do not hear back from. The default status in reporting is ‘not found a job’ even if the person was not job searching. CIRR does allow for programs to report jobs via the grad’s LinkedIn profiles if they were unable to get in touch with them. If no contact was able to be made by the team and LinkedIn shows that the graduate is employed full time at a verified employer, their employment status/title can be included in the report. If the timeline is noted by months on the LinkedIn profile, the start date can be included as the first day of the reported month. No salary information is included if reported via LinkedIn. 

What constitutes ‘Employed in field?

For employment to be validated by CIRR, the graduate’s role must be a paid experience. CIRR’s employment results are categorized as  1: ‘Employed in field’ - split into 1A - Full time employee (30+ hours/week, 6+ months), 1B - Full-time apprenticeship, internship, or contract position (30+ hours/week, 3-6 months), 1C- Short-term contract, part-time position, freelance, or unknown length, 1D - Started a new company or venture after graduation. ‘Employed in Field’ does not include open source work unless paid.

There are also categorizations for graduates who do not seek employment in-field after graduation (2A - employed out of field, 2B - continuing higher education, 2C- not seeking a job for health family, or personal reasons).

The categorization “still seeking a job in field” signifies a graduate of the program is continually looking for a role 6 months after graduating their immersive program.

What happens when someone gets an offer after 180 days? 

Over the years many graduates have obtained full-time offers after the 6-month mark. These graduates are not permitted to be included in the CIRR report. Even if someone’s offer is obtained at 181 days they do not count towards an offer. We saw this happen relatively often in the cohorts that graduated on/around March of 2020, when many graduates obtained offers shortly after 180 days, but could not be included in the CIRR report. While we, of course, want to celebrate these graduates' successes, we always respect the rigorous standards that CIRR has established. 

Who is included in the “hired by school” statistic? 

Codesmith deeply values team members who grow their careers from the ground up and have a first-hand understanding of the mission behind the work that we do. For this reason, we appreciate instructors who have completed the immersive program themselves and understand the student experience. When you see the percentage hired by school on a Codesmith CIRR report, these are a very small handful of graduates that have moved on to work full time as instructors or Software Engineers for the company. Note that fellows are not included in this portion of the report (this is a standard that Codesmith has set within our own reports, and is not a governing standard within CIRR as an organization). This percentage can vary from 1.5 to up to 5% based on our academic team hiring needs.

How are fellows included in the report? 

Codesmith fellows are graduates of the program that stay on to assist the team in core mentorship responsibilities for incoming students, similar to a TA in a traditional university. Fellows are considered to “graduate” the program once they finish their fellowship, typically after an additional 3 months (though some fellows do extend their contract and stay on longer, and a small handful grow to have long-term careers at Codesmith). Despite working on a 3-month contract after graduation, most fellows obtain roles shortly after finishing their contracted positions, and for that reason are included in the report with their original student cohort. 

Can Codesmith remove graduates from the report for low test scores/academic performance?

No, Codesmith does not remove any graduates from the reports based on their academic performance. 

How much of Codesmith’s results were done from fully-remote programs?

Codesmith’s programs have been running fully remote as of March 2020, and any programs completed after that time were done in a fully remote environment. It is worth noting that our part-time immersive program launched in February 2020, and currently has only a few cohorts with published outcomes for our part-time programming. We are excited and eager to share outcomes for other part-time cohorts in the coming months. 

How does auditing work? 

Every single CIRR report submitted by Codesmith is audited by a third party institution. Our audits are conducted by Banks Finley White & Co . CIRR specifies that auditors must be a ‘peer reviewed’ firm to be officially included in the report (https://us.aicpa.org/research/standards/peerreview/peer-review-summary). The firm follows peer-reviewed auditing standards (AICPA AU Section 150 and AU Section 350) to corroborate the outcomes and confirm information is accurately reported.

CIRR requires programs to audit results on an annual basis for the previous 12 calendar months of reports (H1 and H2). From 2022 onwards we aim to submit audited versions within 60 days of initial submission.

What is Codesmith’s relationship with CIRR? 

All member schools are encouraged to nominate a team member to the CIRR board to contribute to its organization’s mission. Codesmith recently joined the board of CIRR (as of May 2022). 

The reporting standards of CIRR were defined in 2016 and were last updated August 10, 2020. 

Standards and the worksheet to develop a report are available on the CIRR website (https://cirr.org/standards). Historical standards are available for comparison.

What does Codesmith see as some of the downfalls of CIRR? 

Although we see CIRR as critical to the importance of accurate reporting, any reporting organization will have its flaws. Outlined here are some of the downfalls of CIRR: 

• CIRR only assess programs that are less traditional (not college/masters degrees) so rather than all professional development programs being assessed against traditional college degrees they’re assessed against the top performing alternative programs.
• CIRR standards can be inflexible when life situations change - for example, someone who decides not to seek employment following graduation but does not announce that before the program commences would be considered to have not obtained employment. While we appreciate the strictness of CIRR’s standards, there are rare cases when the reality of the circumstances might be better served by a bit of flexibility.
• CIRR does not account for additional compensation - a significant portion of many of our graduates overall compensation is outside of their base salary (stock options, signing/annual bonus, etc) 

Does Codesmith track career growth for graduates after their first role? 

CIRR is designed to track outcomes information for graduates only 180 days after completion of the program. With consideration that Codesmith hiring support is for life, we have recently published a report titled Where Are They Now  to document (and celebrate!) the career growth of our graduates over the years. We will continue to publish these reports to track our graduates’ ongoing growth and success. 

How should I interpret CIRR results? 

CIRR results should be interpreted in comparison to a range of sources in education. Graduate schools increasingly provide outcomes insights (e.g. HBS, Yale Law, Stanford Engineering MS).  Results should be interpreted alongside these reports, as well as other CIRR reports, other programs’ reports, and Codemith’s Where Are They Now report (report documenting graduate outcomes and career growth beyond the CIRR standard of 180 days post graduation). 

It is also important to recognize variance year to year. Even in the absence of an actual trend, year-to-year comparisons in time series data will exhibit fluctuations that aren’t indicative of statistically significant changes. This is one of the reasons why statisticians prefer the use of moving averages when analyzing time series data.

Why is the number of graduates included in the report lower than the actual cohort sizes combined? 

The CIRR section documenting the number of graduates in the report is only inclusive of those that intend to seek in-field employment within 180 days post graduation. The field “Graduates Included in the Report” does not include students who did not graduate the program nor students who did not intend to seek employment post graduation.

To read more about CIRR standards, visit: https://cirr.org/standards

If you have any additional questions, email us at hello@codesmith.io to let us know! We’d love to hear from you and will continue to update this article as new questions come in.

Last Updated April 4, 2023