Introduction
Deploying an application to Kubernetes is relatively straightforward when everything works as expected. The real challenge begins when Kubernetes reports that the deployment is running, but the application behaves differently from what the operations team expects.
In enterprise environments, a successful deployment involves more than updating a container image. Applications often depend on external databases, configuration services, network connectivity, persistent storage, and other components that must work together.
During my work with enterprise applications deployed on AWS Elastic Kubernetes Service (EKS), I have encountered situations where a pod was running but the application had not initialized, database migrations failed without affecting the reported health status, and new workloads could not be scheduled despite otherwise valid deployment configurations.
These experiences reinforced an important engineering principle:
Infrastructure health, application readiness, and business functionality are three different things.
In this article, I will discuss several practical deployment scenarios, the investigation approaches used, and the architectural lessons they offer.
The examples are based on real deployment troubleshooting experiences, with infrastructure identifiers and implementation details generalized for confidentiality.
1. When a Running Pod Is Not a Ready Application
One particularly interesting deployment incident involved a Java application that started successfully at the container level but failed to complete its internal initialization.
The new application pod reached the Running state, yet remained 0/1 Ready.
There was no continuous restart loop. The Java process was still active, and several internal health checks were passing.
However, the application never became ready to receive traffic.
Investigation
The investigation started with basic Kubernetes commands:
# Check pod status
kubectl get pods -n <namespace>
# Inspect pod details and events
kubectl describe pod <pod-name> -n <namespace>
# Inspect application logs
kubectl logs <pod-name> -n <namespace>
# Inspect deployment rollout status
kubectl rollout status deployment/<deployment-name> -n <namespace>
The application logs contained an initialization error similar to:
java.lang.IllegalStateException: Channel not active
The error occurred during an internal communication lifecycle callback.
Further investigation showed that the application's database connectivity, memory, and other basic checks were reporting healthy results.
However, the application-server readiness check remained unsuccessful.
The important observation was that the Java process had not terminated.
This ruled out a conventional application crash and shifted the investigation toward the startup lifecycle.
Understanding the failure
The evidence suggested a startup-ordering problem.
An internal lifecycle operation attempted to use a communication channel before it became active.
This was consistent with a race condition during application startup, although the available logs did not establish every aspect of the underlying concurrency behavior.
Because the JVM remained alive, Kubernetes did not report a CrashLoopBackOff.
The container was running, but the application had not completed initialization.
This distinction matters because a container's running state only indicates that its process is executing. It does not automatically prove that all required application components are operational.
Recovery
A controlled rollout restart initiated a fresh startup attempt:
kubectl rollout restart deployment/<deployment-name> \
-n <namespace>
On the next attempt, the communication component initialized successfully and the pod reached 1/1 Ready.
The previous application instance remained healthy during the unsuccessful rollout, so the active service was not interrupted.
However, an important distinction remained.
Restarting the application resolved the immediate operational problem, but it did not establish that the suspected startup race had been permanently fixed.
Architectural lesson
Application readiness must represent the service's ability to perform its intended function, not merely the presence of a running Java process.
For enterprise applications with multiple initialization stages, startup dependencies should be explicit.
When one component depends on another, the application should wait for a reliable initialization signal rather than assuming that the dependent service is ready.
Where appropriate, Kubernetes startup probes can protect applications that require longer initialization periods, while readiness probes determine whether they should receive traffic.
A restart may recover from a transient initialization problem, but repeated restarts should not become a substitute for correcting the underlying startup sequence.
2. When Health Checks Pass but Database Migration Fails
Another incident revealed a less obvious limitation of application health checks.
During deployment, database changes were managed through Liquibase. These changes included configuration schema updates and business data migrations.
In some deployments, the application instances started successfully and reported a healthy status even though one of the database migration operations had failed.
The problem
The migration process encountered errors such as:
Validation Failed:
Checksum mismatch
In other cases, the error was related to existing database records:
ERROR: duplicate key value violates unique constraint
For example, a migration attempted to insert a configuration record that already existed in the target environment.
Liquibase stopped processing at the failing changeset, leaving subsequent database updates unapplied.
However, the application could still establish a database connection and start.
From Kubernetes' perspective, the application appeared healthy.
From a release-management perspective, the deployment was incomplete.
Why this matters
A database connectivity health check generally answers one question:
Can the application establish a connection to the database?
It does not necessarily establish whether:
- All required migrations have completed.
- The database schema matches the deployed application version.
- Required configuration records exist.
- Data transformations completed successfully.
- Business-level operations can execute correctly.
In one deployment scenario, the application image was updated while the business schema remained at the previous version because a migration failed.
The application was technically running, but the application version and database migration state were no longer aligned.
This discrepancy was not visible from the application's ordinary health response.
Investigation
A reliable investigation should examine both Kubernetes and the database migration process.
# Inspect available migration jobs
kubectl get jobs -n <namespace>
# Inspect the migration job
kubectl describe job <migration-job-name> \
-n <namespace>
# Review migration job logs
kubectl logs job/<migration-job-name> \
-n <namespace>
# Review a specific init container
kubectl logs <pod-name> \
-c <init-container-name> \
-n <namespace>
Where Liquibase is used, the database changelog tables can provide additional evidence about which changesets were executed.
A read-only query against a suitably qualified changelog table can help inspect migration history:
SELECT
id,
author,
filename,
dateexecuted,
orderexecuted,
exectype
FROM databasechangelog
ORDER BY orderexecuted DESC;
The exact table location depends on how Liquibase schemas are configured.
These checks should be compared against the migration version and changesets expected for the deployed application release.
Simply finding a database connection or a running pod is insufficient.
Architectural lesson
Database migration success should be a separate release validation requirement.
A reliable deployment process should establish that:
- The correct migration scripts were included in the release.
- The required changesets completed successfully.
- The resulting database state matches the application's expectations.
- No required business configuration was omitted.
- Representative functional operations succeed after deployment.
Where possible, migration workflows should be explicitly coordinated with application startup and release promotion.
Database changes must also be designed with repeatability and idempotency in mind, particularly when different environments may already contain configuration records introduced through earlier operational activities.
A successful readiness response is not proof that database migration completed successfully.
3. Resource Constraints During Kubernetes Rollouts
Another deployment issue occurred when new application pods could not be scheduled because the Kubernetes cluster lacked sufficient resources.
The affected pod remained in the Pending state.
Unlike the previous startup failure, the application process had never started.
Investigation
The investigation began by checking the pod's scheduling events:
kubectl get pods -n <namespace>
kubectl describe pod <pod-name> \
-n <namespace>
The events showed messages similar to:
FailedScheduling:
Insufficient cpu
Insufficient memory
The environment used a small cluster where existing workloads were already consuming most of the available node capacity.
During the rollout, Kubernetes needed to accommodate both existing and replacement workloads.
The additional resource demand exceeded the cluster's scheduling capacity.
Why rolling deployments require spare capacity
Rolling deployments are intended to introduce new application instances while existing instances continue serving traffic.
However, that behavior depends on the available resources and the deployment strategy.
Consider a simplified environment with two existing application workloads.
During an upgrade, Kubernetes may need capacity for one or more additional pods before terminating older instances.
If the nodes are already close to their allocatable resource limits, new pods may remain unscheduled.
The problem can become more visible when blue and green application versions are deployed simultaneously.
Even if each deployment works individually, the combined resource requirements may exceed the available cluster capacity.
Additional diagnostics
Resource investigations may include:
# Review available nodes
kubectl get nodes
# Inspect node capacity and allocated resources
kubectl describe node <node-name>
# Inspect resource requests and limits
kubectl describe pod <pod-name> \
-n <namespace>
# View node resource usage if Metrics Server is available
kubectl top nodes
# View pod resource usage
kubectl top pods -n <namespace>
An important distinction is that the Kubernetes scheduler evaluates resource requests and scheduling constraints, not simply the current CPU or memory utilization shown by monitoring tools.
Therefore, an apparently idle node may still be unable to schedule a pod if its allocatable resources are already reserved by existing workload requests.
Recovery considerations
In non-production environments, temporarily reducing unused application replicas may free enough capacity for a new deployment.
However, this is an operational workaround rather than a long-term architectural solution.
More sustainable approaches include reviewing resource requests, planning node capacity, configuring suitable autoscaling, and defining an appropriate rollout strategy.
Architectural lesson
Cluster sizing should account for deployment-time resource requirements, not only steady-state application consumption.
Before implementing a rolling or blue-green deployment strategy, architects should understand:
- How many old and new pods may coexist.
- How resource requests affect scheduling.
- Whether cluster capacity supports temporary workload overlap.
- Whether autoscaling can provide additional resources.
- How deployment behavior changes when scheduling is delayed.
A deployment strategy that works under normal conditions may fail during an upgrade if spare capacity has not been planned.
4. The Hidden Risk of Configuration Changes
Containerized applications are often described as immutable deployments.
While the container image may be immutable, the configuration surrounding the application is not.
Configuration may originate from Kubernetes ConfigMaps, Secrets, external secret-management systems, application databases, or custom configuration platforms.
In one deployment scenario, rebuilding application configuration data removed values that the application required.
The application image itself was not necessarily faulty.
The problem originated from missing environment-specific configuration.
Understanding the configuration problem
Some enterprise applications store configuration records in database schemas rather than exclusively using Kubernetes ConfigMaps or Secrets.
When deployment procedures rebuild those schemas, configuration that was created or modified manually may be lost unless it is explicitly preserved or recreated.
This creates an important distinction between deploying new application binaries and rebuilding application configuration.
A database schema operation can have consequences beyond the code version being deployed.
Practical example
Consider an application that depends on:
- Database connection configuration.
- External API endpoints.
- Authentication credentials.
- Encryption or key-management references.
- Environment-specific application settings.
If a configuration rebuild removes one of these values, the application may fail during startup or encounter functional failures after becoming ready.
The resulting problem can appear to be an application defect even when the real cause is missing configuration.
Architectural lesson
Configuration ownership and persistence should be clearly defined.
A deployment process should identify:
- Which configuration is maintained through source-controlled definitions.
- Which values are retrieved from external secret-management systems.
- Which configuration must survive application upgrades.
- Which values are environment-specific.
- How required settings are validated before the application becomes available.
Sensitive values should not depend on manual re-entry after normal deployments.
Where possible, configuration should be reproducible, securely managed, and validated through automated release processes.
An immutable container image does not guarantee an immutable or reproducible application environment.
5. Why Cancelling a Deployment Does Not Guarantee Rollback
A more complex deployment incident involved an interrupted Helm upgrade.
The upgrade had already submitted operations to the Kubernetes API server when the client-side process was interrupted.
Although the operator stopped the command, some server-side operations continued.
A database operation completed, while the remainder of the deployment did not finish as expected.
This created a difference between the intended deployment state and the actual state of the environment.
Understanding the problem
Helm manages Kubernetes resources through the Kubernetes API.
When an upgrade starts, resources such as Deployments and Jobs may already have been created or updated.
Interrupting the Helm command does not automatically reverse every operation already accepted by the Kubernetes API server.
For example, a migration Job may continue executing after the command that initiated the deployment has been interrupted.
If that Job modifies an external database, those changes may persist independently of the Helm release state.
Investigation
Before attempting another deployment, the actual cluster state must be understood.
Useful checks include:
# Check Helm release history
helm history <release-name> \
-n <namespace>
# Inspect the current Helm release
helm status <release-name> \
-n <namespace>
# Check Kubernetes Jobs
kubectl get jobs -n <namespace>
# Check pods created by the deployment
kubectl get pods -n <namespace>
# Inspect a particular Job
kubectl describe job <job-name> \
-n <namespace>
# Inspect Job execution logs
kubectl logs job/<job-name> \
-n <namespace>
These checks help determine whether deployment-related operations are still active, completed, or failed.
However, Kubernetes resource status alone may not reveal the full database state.
Migration logs and database changelog records should also be examined before retrying the release.
Rollback considerations
A Helm rollback primarily restores the release's Kubernetes resource configuration to a previous revision.
It does not automatically reverse all database modifications or external side effects caused by the deployment.
For this reason, destructive database operations require particular care.
A cancelled deployment followed by an immediate retry can be dangerous if the first attempt has already partially modified the database.
Architectural lesson
Deployment recovery should begin with understanding the actual state, not simply repeating the previous command.
For complex enterprise applications, recovery procedures should distinguish between:
- Kubernetes resource changes.
- Application configuration updates.
- Database schema modifications.
- Data migration operations.
- External integrations or side effects.
A deployment rollback is not necessarily a business-system rollback.
6. A Practical Deployment Validation Checklist
These experiences changed how I think about deployment verification.
Rather than relying exclusively on a successful Helm command or the readiness status of Kubernetes pods, I prefer to evaluate deployments through several independent validation layers.
Infrastructure validation
Verify that the expected pods are scheduled, storage resources are available, networking dependencies are accessible, and workloads have sufficient resources.
kubectl get pods -n <namespace>
kubectl get deployments -n <namespace>
kubectl get pvc -n <namespace>
kubectl get events -n <namespace> \
--sort-by=.lastTimestamp
Application validation
Confirm that the application has completed initialization and that critical readiness components are operational.
A running container does not automatically indicate that the application is ready to handle business requests.
Database validation
Check that the required migration operations completed successfully.
Verify the database migration state independently from the application's database connectivity health check.
Where appropriate, inspect Liquibase changelog history and confirm that the expected release changesets were applied.
Release validation
Confirm that the running container image corresponds to the intended release.
Review the Helm release status and ensure that no unexpected migration or coordination Jobs remain active.
helm status <release-name> -n <namespace>
kubectl rollout status deployment/<deployment-name> \
-n <namespace>
kubectl get jobs -n <namespace>
Functional validation
Exercise representative application operations after deployment.
For example, enterprise application validation may include authentication, transaction inquiries, API communication, or other critical workflows appropriate to the system.
Infrastructure health checks should complement functional testing rather than replace it.
Recovery validation
Confirm that there is a documented and tested recovery approach if the new release fails.
This may include preserving the previous application version, restoring configuration, or executing a controlled rollback.
Database operations that cannot be automatically reversed should be explicitly documented.
The objective is to ensure that deployment failures can be understood and recovered from without unnecessary additional changes.
7. Key Lessons from Real-World Kubernetes Deployments
Across these incidents, several engineering lessons consistently emerge.
1. Running does not mean ready.
A container can remain active even when important application initialization steps have failed.
2. Readiness does not guarantee business correctness.
A service may pass health checks while database migrations or required configuration updates remain incomplete.
3. Rolling deployments require additional capacity.
Resource planning must consider temporary overlap between existing and replacement workloads.
4. Configuration requires the same discipline as application code.
Environment-specific settings, credentials, and configuration records should be reproducible and safely managed.
5. Deployment interruption does not guarantee rollback.
Kubernetes Jobs, database migrations, and external side effects may continue after the client-side deployment process stops.
6. Immediate recovery and permanent resolution are different activities.
Restarting a pod or retrying a deployment may restore service, but unresolved architectural defects can recur.
7. Deployment success should be validated at multiple levels.
Infrastructure, application readiness, database state, release consistency, and functional correctness require separate checks.
Conclusion
Kubernetes provides powerful capabilities for orchestrating and managing applications, but it cannot automatically determine whether every aspect of an enterprise system is correct.
The incidents discussed here demonstrate how application startup ordering, database migration state, resource constraints, configuration management, and interrupted deployment operations can affect an otherwise routine release.
One of the most useful lessons I have learned from working with enterprise applications on Kubernetes is that a green health indicator is evidence of a particular check passing, not proof that the entire system is functioning correctly.
Reliable deployments require carefully designed readiness signals, explicit migration verification, adequate rollout capacity, controlled configuration management, and meaningful functional validation.
The objective is not merely to make deployments succeed.
It is to make their outcomes predictable, observable, and recoverable.
About the Author
Naveed Akram is a Solutions Architect at Diebold Nixdorf with nearly 16 years of professional experience in software engineering, enterprise architecture, digital banking, and cloud technologies. His technical interests include distributed systems, Kubernetes, AWS, enterprise integration, and AI-assisted software engineering.#kubernetes #aws #architecture #devops #software #coding #development #engineering #inclusive #community
Lessons from Deploying Mission-Critical Applications on Kubernetes: Reliability, Configuration, and Troubleshooting
