Software - There is no box

People think that AI will accelerate software development by generating code. Coding is only a tiny part of the story. So much of software development precedes code, and these activities are usually the blockers that impede technology acceleration.

Before any code can be written, there is the question of what needs to be built. No work can begin until there is a purpose. Work requires time and resources, which necessitates investment. An investment needs a business case. This means getting to know what customers (users) want, what they are willing to pay for, and whether it would be worth building. You are not an AI accelerated software developer, if you have no purpose, if you don’t know what your customers want, or if you can’t justify the investment.

Traditionally, coders working in a commercial setting have relied on product managers, business analysts, and executives to acquire such knowledge and make such decisions. Usually the coders are not subject matter experts. Coders are always asking “what is the requirement?” Coders are paralyzed waiting for answers.

They don’t know the problem domain (the customer’s business) and the terms of art. They don’t have a mental model of the problem space. They don’t know what the actors do (journeys, use cases) and how they collaborate to accomplish their business objectives. They don’t know how these actors do their work, how they need to see their information, and what tasks they perform in certain contexts. They don’t know scale (how many users, how many transactions per hour, how much information of each type). They don’t know cost. They don’t know availability. They don’t know regulatory compliance. They don’t know their customer’s products, pricing, and business policies. There is so much they don’t know about the world, because all they know is how to code.

Coding is always blocked waiting for these responsibilities to be fulfilled by roles who don’t code. The decision makers don’t know the technical side of how to code. The coders don’t know the business side of why or what to code. The collaboration is always impaired by this mismatch in skills and knowledge.

For AI to accelerate software development, the business-focused roles must be elided with the technically-focused role. AI tools must be built to fill the skills and knowledge gap for whomever is in the driver’s seat. Human jobs need to be lost and replaced by AI. Not clear is whether coders will learn to use AI to drive the business; or whether product managers will learn to use AI to code. My bet is on the latter, even though my sympathies and bias are with the former. I don’t see AI gaining the necessary competencies to become entrepreneurial, visionary, and business savvy.

On the other hand, very few people in the business-focused roles are any good at it. We need only look at the lack of success that Tim Cook, CEO of Apple, is having in introducing innovative new products relative to what Apple has done in the past under the leadership of Steve Jobs, a true visionary and a genius in designing desirable products. All lower-ranked roles are responsible for a lesser subset of Tim Cook’s decision-making. It is a rare gem who can achieve 1% the success of a Steve Jobs, when it comes to product vision and design. Coders who have this kind of talent would have become founders. Perhaps, AI can help accelerate this more for coders who have such an inclination. By the sounds of how Tom Bilyeu applies AI for exploring business ideas, this is not as far fetched as I imagine (because of my own ignorance in applying AI in this way).

In the near-term, the quality of vibe code generated by AI is hit-or-miss. A great deal of human supervision is needed to make it work. With the progress we are seeing in AI advances and the cadence accelerating, we should not dismiss the possibility that vibe coding without a heavy human burden for supervision will become routine. Since this is a goal that many people are pursuing, this outcome is pretty much guaranteed. There is virtually no chance it will NOT happen.

Until recently, I did not think that AI would have good taste and judgement to produce good designs. Then, I was introduced to Cline. It is a coding assistant. You guide it by writing a project brief, which documents your project’s purpose, structure, standards, guidelines, constraints, technology selections, style preferences, and rules. You write these in exactly the manner that you normally would in concise English for human team mates. Amazingly, Cline understands and complies perfectly. Through this experience, I am now confident that an AI can be guided to do good design by documenting design principles, archetypes, patterns, and trade-offs. AI may not be there yet today. There is virtually no chance it will NOT happen.

DevOps – development and operations

Developers, who are new to Operations, as they become immersed in DevOps culture, may envision that their involvement in operations follows after development is done. However, operations are not an after-thought. This article is to enumerate some operations-related impacts to development practices that may not be at the forefront of a developer’s mind, but they should be. Design for operations.

Logging to enable monitoring, alerting, troubleshooting, and problem resolution

When coding and testing, pay attention to how helpful log messages are to troubleshooting problems. Do messages contain enough context to assist in identifying the root cause and corrective actions? Are messages at appropriate severity levels? One of the biggest impediments to monitoring and troubleshooting is excessive, imprecise, and unnecessary logging, otherwise known as noise.

If logging an ERROR level message, it should represent an operational problem that can be monitored. Each error should have a corresponding corrective action, if one is necessary. If a failure is transient and correctable by retrying, It should not be logged as an error until all attempts have been exhausted without success; repeated messages are unhelpful. Error level messages must be documented in the monitoring, troubleshooting, and problem resolution procedures. Error messages that are functional without any operational significance (not correctable through operational procedures) should be marked as such, so that they are not monitored for intervention. The knowledge base should document every foreseeable failure mode and its corrective actions.

Every log message incurs cost.

Computational cost to produce the message and collect it.
Storage cost for retention and indexing for search. Accounting for the volume of messages collected per service instance per day multiplied by one year retention and the number of service instances deployed, that may be hundreds of terabytes of data at a cost of tens of thousands of dollars per month.
Documentation cost to understand the meaning of the message and the expected operational procedures, if any, to monitor, alert, and carry out any corrective actions, when detected.

Excessive logging produces noise that becomes an impediment to operational efficiency and effectiveness. Monitoring and troubleshooting become more difficult, when significant information is buried among the noise. Seek to reduce noise by eliminating log messages that are not valuable. This can be done by classifying messages at a finer-grained log level (i.e., INFO, DEBUG) or by suppressing them altogether.

Specify a log message format so that alerts can be defined based on pattern matching. A precise identifier (e.g., OLTP-0123) for each type of message is helpful for monitoring solutions to key off of, rather than matching arbitrary strings that are not guaranteed to remain invariant.

Log messages should be parameterized to carry contextual information, such as the identifier of the entity being processed and the values of the most significant properties to the transaction. Avoid logging sensitive data that would compromise security, such as credentials or personally identifiable information subject to data privacy regulations. The context is important for isolating the problem, when trouble shooting, parameterizing corrective actions to resolve the problem, and providing a useful description when reporting a bug.

Avoid logging stack traces for non-debug levels of logging. Stack traces are verbose (noisy), and they carry information about the internal workings of the software (packages, classes, and source file names) that may be interesting to developers, but is not useful for operations.

Avoid repeatedly logging the same message. Repeated logging is often the consequence of an error condition that is handled with a retry loop. To avoid noise, retries can be counted without logging the continuing error. A summary can be logged when the retry loop ends. If retrying is successful, silence is preferred unless it is important to note violations of performance targets caused by retrying. Otherwise, timing out may entail escalating the error condition to a fall back mechanism or a circuit breaker, and logging this exceptional condition may be informative later, if the condition persists.

Do log a message when an error is detected for which a bug should be reported, an operations engineer should be alerted about a possible malfunction, or a corrective action is required. Error conditions that represent a possible service outage are especially important, as these are the messages that should be matched for alerting. Errors will be associated with corrective actions, which operations engineers will perform to resolve the problem, when encountered.

Avoid logging a message for normal operations, such as successful liveness probes and readiness probes. This is worthless noise.

Pay special attention to methods and transactions with security events.

Redirect a request to access a user interface to login
A successful login
A failed login attempt
Performing a privileged action that must be audited, such as administrative actions or gaining access to private information not owned by the user
A denial of access due to insufficient privileges

Security events should be logged with a format that allows such messages to be classified, so that they can be directed to SIEM for special handling. SIEM is responsible for auditing, intrusion detection, and fraud detection. Being able to detect a security breach is among the most important operational responsibilities.

Audiences

One of the most common mistakes is to conflate the log stream directed toward operations and information intended for end users. Services that enable administrative end users to configure or customize features need to provide transparency to the steps that are executing. This includes integrating other services to collaborate or specifying workflows or policies, which can be done erroneously. When something can go wrong, users must have a way of debugging those errors. Legacy server-based applications tended to not differentiate between these use cases and operations. Unfortunately, when evolving such applications into cloud services, these use cases are the most difficult to tease apart, so that information is directed to the proper audiences.

It is equally problematic when functional issues, which are intended for feedback to end users, are directed to operational logs. Operations have no role to play in monitoring such issues and taking corrective action. Directing such messages to operational logs adds to noise.

Developers must pay attention to the intended audience of messages, so that they are either directed to operational logs or end users or both.

Telemetry

“You get what you measure.”

Produce metrics for things you care about.

Service utilization – end user activity
Service outages – times when the liveness and readiness probes fail, so that these outages go toward calculating the Recovery Time Objective (RTO) and Recovery Point Objective (RPO) for service availability
Latency – the time between receiving a request and sending its corresponding response tell us how well the system is performing, as perceived by human users interacting through a user interface
Transaction processing throughput – the volume of transactions processed in each time interval tell us how well the system is performing with regard to work loads
Resource utilization – the infrastructure and platform resources (compute, storage, network) consumed to provide the service tell us how the demand is trending and how capacity should be managed to enable the service to scale into the future

Collect these metrics in a time series database for monitoring (including visualization and reporting) and alerting based on threshold crossing. Specify threshold crossing events for conditions that need attention, such as the following.

Violations of service level objectives (availability, performance)
Exceeding service limits (what the user is allowed) toward up-selling higher levels of service to the customer
Exceeding service and resource demands (capacity planning) toward adjusting scale and forecasting future scaling needs

Autonomous operation

Developers should design a service to be self-sustaining indefinitely without the need for human intervention, as much as possible. Human involvement is expensive (labor intensive), error-prone, and slow to respond compared to automated procedures. Any condition that resorts to human intervention should be considered a failure on the part of the developers to design for autonomous operation.

Fault tolerance – as a consumer of other services and resources, be resilient to failures and outages that are likely to recover after some time
Self-healing – as a provider that is experiencing a failure or outage, detect the problem and implement measures to recover, such as restarting, rescheduling to use alternative resources, or shifting workloads to surviving instances
Auto-scaling – adjust the number of resources to match the workload, so that the service continues to satisfy performance objectives
Self-maintaining – routine housekeeping should be scheduled and automated to prevent storage exhaustion (i.e., data purging), to maintain reasonable performance (i.e., recalculate statistics), and to enforce policies (i.e., secrets rotation)

Pipelines

Operations involve actions initiated through human access (by operations staff) and systems integration (i.e., by capturing an order submitted by the subscriber). Ad hoc changes to a production deployment should be forbidden, because these cannot be reproduced programmatically from source code. Therefore, all types of changes must be anticipated during development, so that they are available as pipelines (programmatic workflows) operationally. Each action should be parameterized, so that a precise set of information is input to drive the execution of its pipeline.

Provisioning – creation and termination of the service subscription

Upgrades and patches – deploying software updates and bug fixes

Configuration – scaling, enabling and disabling features, naming, certificates, policies, customization

Diagnostic actions – checks, tests, and probes with a verbose log level for troubleshooting and debugging, when increased scrutiny is needed for problem resolution

Administrative actions and maintenance – password and secrets rotation, data purging according to retention policy, and housekeeping (e.g., storage optimization)

Capacity management – adding and removing infrastructure and platform resources (e.g., compute, storage, addresses)

Corrective actions – interventions for problem resolution, such as stopping and restarting of services (rescheduling on compute resources), forcing maintenance tasks like purging data (due to storage exhaustion) or password rotation (prevent security breaches), replacing defective resources (e.g., kernel deadlocks)

There is no box

Tag Archives: Software

Use AI to Accelerate Software Development

Insights into innovation