Sema Blog

Developer Coaching and Evaluation Principles

Posted by Sema on Apr 28, 2019 4:02:17 PM


Sema is founded on three sacred trusts to engineers, organization leaders and other members of the software community. All three are essential to carrying out our work.

  • Protecting our clients' intellectual property.
  • Serving the computer science and research communities by continuously improving how we measure software quality.
  • Contributing accurately and ethically to developer self-improvement, coaching and evaluation.

This post goes deeper on the third point:

Our unequivocal commitment to contributing accurately and ethically to developer self-improvement, coaching and evaluation.

Sema understands that our code quality analytics will be used to evaluate developers in high-stakes situations such as performance reviews and, far more commonly, low-stakes situations such as self-guided improvement and manager-assisted coaching. More just expecting this, we actively welcome it.

We fundamentally believe that bringing code quality data to the process of coaching and evaluating developers is the right thing to do for everyone: developers themselves; the organizations they work for; the software computer science and engineering community as a whole; and the end users of the software that they build.

To participate ethically in this endeavor, we must make our contributions to developer coaching and evaluation the best they can possibly be, and implement our contributions with context awareness, discretion, and empathy.

Developer Coaching and Evaluation Principles

Developer evaluation shares the limitations of code evaluation.

This is tied to the core Sema Principles: our code evaluation must be the best quality possible, and its limitations must be recognized. For example, code quality does not measure its success in meeting functional requirements.

Code evaluation does not tell the full story of a developer’s contribution.

Sema measures only the quality of developers' contributions to the codebase.

We do not measure other essential elements of a developers’ contributions to an organization: mentorship of others, idea generation, fellowship, and other non-development work.

Therefore, Sema is only a contributor to the evaluation of developers as employees and colleagues. In other words, Sema provides an assessment of the developers’ contributions to the code, not a full personnel review.

When an organization is interested in a comprehensive assessment of a developer's value to the organization, it must take into account multiple forms of data. These might include survey data, interviews, or other observations. Sema’s developer evaluation solution only considers that which is measurable within the code and test coverage, and not by these other mechanisms.

As a team of practitioners and advocates of the craft and the profession of software development, Sema has a point of view on the right ways (and the wrong ways!) to conduct comprehensive, integrated developer coaching and evaluation systems. Hint: great systems share fundamental principles – such as a heavy emphasis on self-improvement, followed by coaching, with minimal weight given to formal evaluation – but the specifics vary widely based on the context.

Many excellent examples of other components of great coaching and evaluation systems are already available. Sema focused our contribution to an unsolved problem: adding code quality measures to developer evaluation.

Impact on code quality is a meaningful component of developer coaching and evaluation.

While we do not provide a full personnel review, Sema provides an assessment of the developers’ contributions to the code, and we believe that evaluation is an appropriate component of developer coaching and evaluation.

For example, if code uses really good algorithms and both the data structures and algorithms are optimized for performance, this implies that the developer has thought about important design elements. This developer is also more likely to be thinking about other elements of great code, such as redundancy, repetition, and security.

Developers must have access to tools and information so they can improve their craft.

Today, almost every developer lacks ready access (or any access) to usable, digestible, comprehensive data about how they are changing or impacting code quality. The first step in the journey of developer coaching and evaluation is to give developers access to this data.

The second step which we expect and encourage is self-exploration by those developers to understand their contributions, to experiment, and to measure the impact of their work over time.

The third step is coaching, where a mentor or manager gives directive advice or suggestions on how a developer should change their behavior.

The final step is evaluation. Developers must first have access to usable tools, have had a chance to understand, internalize and use the data, and have received coaching. Only then do we recommend using code quality data as a component of developer evaluations.

Developers have a professional obligation to improve their craft. That is why they are professionals.

Sema accepts that not every developer will be satisfied by our approach. We aspire to satisfy every developer that this approach is not only better than today's state but genuinely good.

We recognize that we will not convince everybody, but we commit to using our best efforts to help developers to see it as we see it: that what Sema is building represents a fundamental step forward for the software engineering profession by giving engineers the right tools to inform and improve their practice.

Sema believes that developers have an obligation to continuous self-improvement. That is non-negotiable for us. Lawyers, teachers, and doctors have an obligation to continue to get better at their craft. Likewise, software engineering is a profession that comes with professional obligations and responsibilities, and that includes doing right by your end user.

We are doing our best to build our contribution to an evaluation system that engineers will like. But we're not just serving engineers: we are helping developers better serve the end users of your code. You may not always love what the data tells you, but you have an obligation to get continually better at your craft.

Coach before you evaluate.

Sema strongly recommends that organizations coach developers towards better performance before beginning evaluation.

We believe that measuring developer contributions to code quality is an essential ingredient in making code better and organizations stronger. We believe this data can and should be used both for coaching and evaluation purposes. However, we believe that outside of unusual circumstances, the first step should always be coaching. We think this is the best outcome for the organization and the most appropriate way to support the professionals who work at the organization.

Sema’s methodology.

Sema has an obligation to share the theory behind our methods and metrics, and a responsibility to explain how those different components are aggregated into a picture of developer contributions. Therefore, we will be transparent on our concepts and the aggregation of our methods, but the specific methods used in our solutions are proprietary to Sema and will not be shared.

How Sema puts these principles to work.

The first step is for developers to have access to Sema's solutions, such as Sema Maintenance Platform so they can understand the impact of their code.

Second, developers use the tool so that they can experiment with it and understand their impact.

Third, Sema's solution is used as the basis for coaching. That coaching is inquiry-based and two-way. For example, “help me understand the tradeoffs you made about the code, and the impact on functional and nonfunctional requirements.”

Fourth, Sema's solution is used as a component of evaluations, but only as one contributing element, supplementing code reviews, interviews, and other methods.

Topics: Developer Evaluation, Developer Coaching and Evaluation Principles, Developer Coaching