Navigating Cloud Migration: Overcoming Challenges and Maximising Value

In today's digital era, where innovation and competitiveness are paramount, cloud migration has become a critical step for organisations embarking on their digital transformation journey. The unparalleled scalability, agility, and cost-efficiency offered by cloud computing have captured the imagination of tech leaders worldwide. However, navigating the complexities of cloud migration requires careful planning and consideration. This article explores various aspects of cloud migration, drawing on battle-tested experiences and research to provide strategies and guiding principles that address challenges and maximise the value of cloud migration initiatives.


Cloud adoption has witnessed a significant surge, with nearly 80% of organisations adopting or planning to adopt a cloud-first strategy, according to Gartner's latest survey. However, despite the growing popularity of cloud services, not all cloud migration efforts are successfully. In fact, around 50% of cloud migration initiatives experience failure according to the same study. A report by IDC highlights the need for decision-makers to address these challenges to ensure a successful migration process and manage post-migration expenses effectively.


This article guides technical and business decision makers through the cloud migration landscape. It covers essential topics such as assessing cloud platforms, designing and developing migration strategy and plan, evaluating cloud migration tools, verifying migration results, monitoring key metrics, and optimising cloud spending (FinOps). By following best practices and leveraging insights from real-world experiences, organisations can navigate the cloud migration journey with confidence, ensuring a seamless transition to the cloud.


Join me as we delve into the intricacies of cloud migration, sharing practical advice and lessons learned, and empowering you to overcome challenges, seize opportunities, and unlock the full potential of cloud computing.

Build a Minimum Viable Product (MVP)

Before investing in any serious migration exercise, you should focus on understanding the technologies, process and toolings involved in the migration process. This will allow further improvements and optimisations prior to large scale rollout across the organisation.


You can start with a pilot programme to experiment with a minimum viable product to gain a better understanding of cloud environments, its offerings and relevant tools. The pilot should be conducted with the following criterias:

  • Defined scope, goals and objectives: agree on a clear roadmap and timetable of the pilot to make sure the outcomes are reliable, measurable and reproducible.

  • Structuring the team appropriately: ensure the team has the right mix of people, expertise and experience to conduct the experiment.

  • Removing impediments: context switching can be painful and costly especially during thai exercise, make sure the team stays focused and is protected from noise so that the project can progress smoothly.

  • Improving continuously: check milestones and feedback regularly to align with set goals and objects, and pivot accordingly.

Establish Migration Strategy


The purpose of the migration strategy is to set the guiding principles and standards for cloud migration. It will focus on determining the business and service level measurements, key performance indicators. A good migration strategy will not only help organisations and teams understand the current landscape better and perform the migration in a safe and controlled manner, but also make sure the services do not deteriorate post-migration.


Prior to large scale migration, you should invest in gaining sufficient understanding of the current state of systems, including performance targets, SLA, dependencies etc. Moreover, you should not overlook processes like data backup and recovery processes for mission critical databases. Any oversight at this stage could result in insufficient resource allocation or waste during the migration, or worse. To set up an effective migration strategy, you need to assess the following aspects:

  • Collect database information, that includes operation model, database technology, database connection, etc to set guidance for provisioning cloud resources.

  • Investigate the current database backup and restore plan, including implementation methods, frequency etc, and make sure they are executable and up-to-date. This should be evaluated for replacement or replication on the cloud once migration is completed.

  • Record key performance indicators of servers and services, such as CPU and memory usage, average IOPS, disk usage, etc to provide a basis for evaluation post-migration.

  • Determine systems dependencies and decide the sequence of migration to reduce impact.

  • Design and develop a testing strategy to ensure business stability throughout and post migration.

Develop Migration Plan

Following the migration strategy, the migration plan sets more detailed guidelines on the system level. The plan should iron out the functional and non-functional requirements of current systems, to set precedents for systems, whether it’s a “lift and shift” or redevelopment. These characteristics should be reflected on the design of cloud infrastructure.


On top of the mentioned areas, the approach to plan your migration matters a lot. There are a number of key considerations when developing the plan:

  • Customise your cloud migration plan. Despite the tooling and instructions out there, there is no one-size-fits-all migration plan. Every organisation has their unique set of applications, systems and infrastructures, and their architecture, performance characteristics and security considerations also vary massively. Therefore, you need to tailor your plan to the characteristics of your systems, and don’t forget to plan for risks associated with migration, including sequence of migration, testing and verification plan, backup and restore plan, etc.

  • Automate tools and processes. Migration process can be prolonged and laborious, using automated tools and processes can massively reduce human error and improve efficiency during migration. For instance, you should invest in infrastructure as code, automated migration scripts, test automation, etc.

  • Security and Compliance. There could be dire consequences if the security and compliance were overlooked. For example, how do you secure your data and prevent it from being tampered or leakage before, during and after migration; how the cloud infrastructure is provisioned in a fashion that remains compliant with local regulations and standards, such as GDPR, HIPAA, etc.

  • Network setup. Network is the gateway to secured and functioning systems, it also serves the monitoring and throttling requirements. For example, network settings control whether systems are exposed publicly or internally; whether those requirements are up-to-date and compliant with company policies.

  • Make a phased migration plan. Instead of risky big-bang migration, it is sensible to have a phased plan to migration that releases changes in incrementals. You can use the strangler pattern when developing the plan, and validate the result of each stage before moving on.

Evaluate Cloud Migration Tools

Part of developing a cloud migration plan is to understand and sharpen the skills on migration tools. If every step before this point has been more analytical, this is where the engineering effort takes place, and stakes get higher. Investing in the right toolkit can decide whether the migration is successful and how successful it is. 


Luckily, mature cloud vendors typically provide a set of tools to assist cloud migration from either on-premises or other cloud vendors, which can help speed up the process and reduce costs and risk. This is how you can measure the effectiveness of the outcome:

  • Understand the technical “nitty gritty” of tools. In the case of container migration, how the tools manage the image synchronisation and directory structure is an important question to answer. For databases, you need to make sure the tools can support the synchronisation cadence and transmission protocols. For services, you need to understand how to replicate the configurations using the provided tools.

  • Consider the limitations of the selected tools. Each migration tool would have its own limitations. Rather than being blindsided by those when it comes to implementation, evaluate the tools to understand their features and capabilities, and make sure those limitations will not slow down or block the migration process.

  • Provide training to the teams. No one knows the systems better than the teams themselves, it is important to empower the team with adequate training when they embark on the migration journey, whether it’s hands-on skills or theories.

  • The migration plan is designed in relation to the selected tools and the systems and well communicated with other teams. Make sure each team has the responsibility and autonomy to migrate their systems, coordinate and facilitate good communication amongst teams and groups.

Verify Migration Result

Migration is not done until you can confirm every system is in full operation. The goal of verification is to ensure the migration is successful and secure, and deal with raised issues immediately. At this stage, the data points collected during the strategy and plan setting stages will be used extensively, this mainly covers the following three areas:

  • Data: Depending on migration methods, whether it’s partial or full migration, make sure the data structure, mapping, filtering, functions are fully replicated; confirm the delta synchronisation is working correctly between the source server to the target cloud server or service; for data in transit, such as Kafka migration, check for the data in the queue has been replicated successfully.

  • Environments: make sure the correct version of services are installed on the cloud servers, verify configurations and data are also replicated to the new environments.

  • Operations and maintenance: monitor relevant SLAs / SLOs to make sure the state of the systems are fully compliant with the expected standards.

Monitor Key Metrics

If monitoring and observability was not your strongest suit, cloud migration provides a unique opportunity to up your game, with the majority of tooling are native to the cloud vendors and integration provided with the cloud services. The benefit of comprehensive monitoring is to provide better insight into your systems, allow faster feedback loop, and improve on-going support and maintenance. Monitoring, reporting and altering should be set up for technical indicators, which I will elaborate below; as well as operational metrics, such as the four-key metrics:

  • Network performance: Monitor traffic and latency, set up rate limiting to make the network more stable.

  • Application performance: This includes health check and functional performance indicators, such as response time, throughput, error rate, etc., as well as the status of the cloud resource, such as CPU, memory, disk space, etc.

  • Security: Practice intrusion detection, vulnerability scanning, log audit, etc., to ensure system security.

  • Costs and resource usage: Monitor cloud resource usage to maximise resource utilisation and reduce unnecessary spending.

  • Four-key metrics: This includes lead time for changes, change failure rate, deployment frequency and mean time to recovery.

Optimise Cloud Spending and FinOps

There could be many incentives to move to cloud, however, it all comes to return on investment and long-term cost at the end. More often than not, what we have seen is engineering teams as well as business tend to treat cloud migration as a one-off exercise, naively thinking the business can go back to normal once the migration is done, this is far from ideal. Cloud migration not only requires miraculous implementation and execution, but also a long-term vision on what cloud means, what it brings and how it benefits the organisation. There is good evidence that companies practising FinOps gain better control over their cloud spending, maximising cloud resource utilisation, and optimising business processes and operational efficiency.


FinOps can be broken down into the following areas to ensure better transparency of the cloud spending:

  • Budget planning: Cloud migration can incur substantial costs and spending, such as  data centre relocation, application migration, training and support, etc. Understanding the costs associated with migration and the on-going cost can help forecasting and budget planning, gain better control and manage spending and avoid unnecessary expenditures and expenditures exceeding budget.

  • Cost control: Post cloud migration, cost structures and billing models can vary massively compared to running on-premise infrastructure. By implementing cost control strategies, you can gain better control of costs, and optimise billing models and resource allocation based on actual usage.

  • Resource optimisation: Resources on the cloud are highly elastic and scalable, which is a double-edged sword. Having a strategic view over your estate, backed with strong data, can help you identify where the waste is happening and where re-prioritisation should take place to improve the overall efficiency.


Take AWS for example, companies can optimise costs in the following ways:

  • Use appropriate dedicated computing resource, i.e. EC2 instances. Select the appropriate instance type and specification according to specific business scenarios and workloads. For example, medium instances are suitable for scenarios such as websites, web applications, or medium-to-light-load applications, while high-performance computing/storage-intensive/GPU instances are suitable for It is used in high-performance computing, video encoding, 3D rendering and other scenarios.

  • Consider different service models, i.e. dedicated server or serverless. The popular on-demand by-usage serverless option, such as Lambda, is better suited for short-term spurs scenarios. Whereas the by-instance subscription model, such as EC2, is more suitable for services that need to be constantly running that require high availability and low latency.

  • Compare different cloud services for the same capability. Take containerised services for example, the containers can run on a wide range of AWS services, such as ECS, EKS and Fargate. You may be exchanging convenience with vendor lockdown in choosing among those options, the services are priced at different points, and they may vary in maturity and community support. Therefore, make sure you weigh up the pros and cons before you make a commitment.

Summary

Cloud migration is not a walk in the park, however, it doesn’t have to be like challenging Mount Everest either. Unsuccessful cloud migration tends to share similar traits: lack of planning, technical challenges, poor change management, overrunning costs and oversight on security and compliance.


However, through good vision and communication, careful planning and execution, and working closely with qualified cloud migration partners, it’s only a matter of time you’d find yourself reaching the other side with a breeze. 


If you have any questions or need specific advice, feel free to reach out to me on Twitter or LinkedIn.

Comments

Post a Comment

Popular posts from this blog

How to: Add Watermark to PDFs Programmatically using iTextSharp

Being able to add text or image watermark to PDF files is one of those handy things you can archive via program. The only library required for the little program is iTextSharp . For those who haven't used it yet, iTextSharp is a C# version of iText , which allows you to generate or manipulate PDFs. So let's cut the chitchat, and go straight to the code. By calling 'AddTextWatermark' or 'AddImageWatermark' function with required parameters supplied, you are ready to rock! public void CreateTemplate(string watermarkText, string targetFileName){     var document = new Document();     var pdfWriter = PdfWriter.GetInstance(document, new FileStream(targetFileName, FileMode.Create));     var font = new Font(Font.FontFamily.HELVETICA, 60, Font.NORMAL, BaseColor.LIGHT_GRAY);     document.Open();     // Specify font and alignment of text watermark     ColumnText.ShowTextAligned(pdfWriter.DirectContent, Element.ALIGN_CENTER, new ...
Read more

A practical guide to Scala Traits

Trait - has a super class of AnyRef   - uses extends  or with for inheriting a trait ( with to inherit multiple traits)   - defines a type   - can be overridden using override  keyword   Trait vs Interface (Java)   - trait can declare fields and main state   - rich vs thin interfaces (every concrete method added to trait makes it richer)   Trait vs Class   - trait can NOT have any class parameters        trait doesn't support coding in this style: class Point(x: Int, y: Int) - super calls are statically bound in classes, but dynamically bound in trait         super.toString  is predicable in classes, but not in trait when mixing   Ordered Trait   Traits as stackable modifications   - order of mixins: right most trait takes effect first   When to use Trait?   - If the behaviour will not be reused, then make it a concrete class   - If it might be reused in multip...
Read more

A Short Guide to AWK

AWK is built to process column-oriented text data, such as tables. In which a file is considered to consist of N records (rows) by M fields (columns) Basic # awk < file ‘{print $2}’ # awk ‘{print $2}’ file file = input file print $2 = 2nd field of the line, awk has whitespace as the default delimiter Delimiter # awk -F: ‘{print $2}’ file -F: = use ‘:’ as delimiter # awk -F ‘[:;]’ ‘{print $2}’ file -F ‘[:;]’ = use multiple delimiter, and parse using EITHER ‘:’ OR ‘;’ # awk -F ‘:;’ ‘{print $2}’ file   -F ‘:;’ = use multiple delimiter, and parse using ':;’ as THE delimiter # awk -F ‘:’+ ‘{print $2}’ file   -F ‘:’+ = use multiple delimiter, to match any number ':' Arithmetic # echo 5 4 | awk '{print $1 + $2}' output is '9', the result of ‘+’ (works as addition) # echo 5 4 | awk '{print $1 $2}' output is '54', to get string concatenation #  echo 5 4 | awk '{print $1, $2}' output is '5 4',...
Read more