## What is AppCapsule and what problems does it solve? 
AppCapsule creates a snapshot of all the data in memory, and all the application states required to restore the application at this moment in time, as well as the data on storage. With AppCapsule, we can transport the application runtime across different machines. 

Three problems that AppCapsule can help to solve - 
1. **Spot** reclaim - If there is a spot eviction event, Memory Machine Cloud creates an AppCapsule, and moves it to another spot instance, where the job will continue. That means, you can save cost running jobs on spot instance, while keeping all your job progress and does not need to run from the beginning all over again. A.k.a. *SpotSurfer*.
2. **OOM** (Out Of Memory) - Memory Machine Cloud detects if OOM is about to happen and if so, creates an AppCapsule, and moves it to another instance to avoid the application crash as well as have the job continue running.
3. **Right-sizing** - vertical scaling down to avoid over-provisioning or up to provide more resources to the application as it is needed for shorter wall-time and possibly lower total cost. You can trigger either manually by using float cli command / OpCenter Web GUI or automatically done by Memory Machine Cloud (a.k.a. *WaveRider*).
 
![](https://baywatch-api.memverge.com/media/upload/202312/231220190505523032.png)

## What is AppCapsule++ and what benefits does it bring? 
Three enhancements and benefits that AppCapsule++ brings to the users - 

1. Only saving **delta** - instead of saving a full snapshot, AppCapsule++ only saves the difference (memory change) since the previous snapshot. This way, the time required for the final snapshot is minimal.
2. **Adaptive** trigger - instead of snapshotting in a fixed, pre-determined interval, AppCapsule++ automatically and intelligently checks the memory change and take a snapshot when it reaches a threshold, which is calculated based on the cloud service provider’s pre-emption window. The benefit is no need to guess the fixed interval for periodic snapshot.
3. **Asynchronous offloading** - instead of synchronous snapshotting which will pause the application, AppCapsule++ does asynchronous data offloading, which decreases the time that the application is frozen. The benefit is lower impact to job completion wall-time. 

Below is an overall comparison table of AppCapsule and AppCapsule++
| Comparison | AppCapsule | AppCapsule++ |
| - | - | - |
| **Snapshot content** | Full snapshot | Delta snapshot |
| **Snapshot trigger** | Can be - Periodic done by MMCloud, or - Manual done by user’s script | Adaptive | 
| **Configuration** | Pre-determined interval for periodic snapshot | No special configuration | 
| **Application completion wall-time** | Impact higher (Synchronous) | Impact lower (Asynchronous) | 
| **Storage overhead** | Relatively lower (only need to save final snapshot to restore) | Relatively higher (need to save all delta snapshots to restore) | 
| **Usage** | `--dumpMode full` | `--dumpMode incremental` |


## Benchmark comparison 
Below is the benchmark experiment result running on AWS comparing AppCapsule and AppCapsule++. The experiment is based on 50GB memory change in terms of 60GB, 120GB, 240GB, and 480GB application memory usage, and how much time in seconds did AppCapsule and AppCapsule++ take to take the final snapshot. 

Three observations from this benchmark - 

1. For AppCapsule, as application memory usage grows, the snapshot time also **increases**. The reason is that AppCapsule takes a complete snapshot every time.
2. For AppCapsule++, as application memory usage grows, the snapshot time **stays more or less the same**. The reason is that AppCapsule++ takes a delta snapshot, which is triggered when the amount of change in memory reaches a certain threshold.
3. Considering using spot instance on AWS, which has 2-minute pre-emption time window, AppCapsule can only support relatively small workload (in this experiment for example, 60GB since the snapshot time is 109 seconds). AppCapsule will not be able to support bigger workload (in this experiment for example, 120GB needs 250 seconds to finish a snapshot which is way over the 2-minute time window). However, since AppCapsule++ only takes delta snapshot, the snapshot time is consistent with memory change, not the application memory usage. 
![AppCapsule vs AppCapsule++](https://baywatch-api.memverge.com/media/upload/202312/231220185252866864.png)


More detailed information in table format below: 
| Experiment No. | Experiment Variable | AppCapsule | AppCapsule++ |
| - | - | - | - |
| 1 | Application memory usage: 60G Application memory change: 50G | 109s r5.2xlarge/gp3 | 91s r5.2xlarge/gp3 |
| 2 | Application memory usage: 120G Application memory change: 50G | 250s r5.4xlarge/io1 | 102s r5.4xlarge/io1 |
| 3 | Application memory usage: 240G Application memory change: 50G | 533s r5.8xlarge/io1 | 104s r5.8xlarge/io1 |
| 4 | Application memory usage: 480G Application memory change: 50G | 1548s r5.16xlarge/io1 | 80s r5.16xlarge/io1 |

## Call to action - save your spending and time 
What is the characteristic of your running job and environment? AppCapsule++ is available now with Memory Machine Cloud 2.4 Goa release. Please reach out or leave comments below and we would love to chat and help!

Anyway you look at it, the Chinese mitten crab (also known as the Shanghai hairy crab because of the setae on its claws) has something to offer. For the chef, a seasonal delicacy — full of sweetness and umami flavors. To the aquaculturalist, an important crop and revenue source. This crab is a marvel to biologists — it can live in freshwater and saltwater? Regenerate limbs? Really?

For the geneticist, the challenge is to trace these phylogenetic behaviors (behaviors typical of the species) back to the genome and transcriptome level.

![](https://baywatch-api.memverge.com/media/upload/202311/231127195706586901.jpeg)

## Eriocheir sinensis

The Chinese mitten crab (*Eriocheir sinensis*) is native to China where it is found in lakes and rivers. In Europe and North America, it is considered an invasive species. The mitten crab spends most of its life in freshwater although the reproductive phase occurs in brackish water and in seawater (the species is catadromous). The change in salinity requires a complicated mechanism (osmoregulation) to maintain the ionic balance in its cells.

The first discovery of Chinese mitten crabs in North America occurred in the San Francisco Bay in [1992](https://wildlife.ca.gov/Conservation/Invasives/Species/Mitten-Crab). Since then, specimens have been found in the Chesapeake Bay, Hudson River, and the Great Lakes. They have been found all over Europe — in the Thames River, the Tagus River, the Danube, and even in the Venice lagoon, although finding an isolated specimen does not mean that a permanent population has established itself. Mitten crabs have a high reproductive rate and can withstand physiological stress (such as starvation and dessication) which makes them a formidable invasive species whose burrows undermine river bank stability. It's a mystery why no Chinese mitten crabs have been seen in California since [2010](https://www.smithsonianmag.com/blogs/smithsonian-environmental-research-center/2018/06/05/wanted-chinese-mitten-crabs/).

Beyond its culinary, agricultural, and environmental impacts, the Chinese mitten crab is worthy of study because of its ability to [regenerate](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9481118/) an entire limb (as long as it can molt). Regeneration capability is stronger in inverterbrates than in vertebrates where it seems to have declined. Birds and  mammals both demonstrate some cell and tissue regeneration capability, so it is an interesting topic of medical research to see if regeneration ability can be stimulated.

It is no surprise, then, that *Eriocheir sinensis* has been studied extensively, in particular, to discover the gene expressions that underly its physiological and behavioral traits.  The first genome sequencing, assembly, and annotation of the Chinese mitten crab was reported in [2016](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4730596/) and the current reference assembly was posted in [2022](https://www.ncbi.nlm.nih.gov/datasets/genome/GCF_024679095.1/). The haploid [chromosome number](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4814232/) is 73 (2n=146) and the genome size is about 1.27 Gb, which puts it the high end of chromosome number among [arthropods](https://www.nature.com/articles/s41467-021-22604-3) (inverterbrates with exoskeletons), although there is a large percentage of repetitive elements. For comparison, the haploid chromosome number for humans is 23 and the haploid genome has about 3.1 Gb.

Nextflow and MMCloud
--------------------

So, what has this to do with Memory Machine Cloud (MMCloud)? RNA-seq is a popular computational pipeline for analysing gene expression and transcription activation using data generated by next-generation sequencing (NGS) machines from Illumina, PacBio, Oxford Nanopore Technologies, and others. A complicated pipeline, like RNA-seq, requires a workflow manager, such as Nextflow, to schedule and manage the individual tasks in the pipeline.


In Nextflow terminology, each task is assigned to an "executor," which is a complete environment for running that step in the analysis. By attaching the [nf-float](https://github.com/MemVerge/nf-float/blob/master/README.md) plugin to a workflow, Nextflow can use MMCloud as an "executor." From an MMCloud point of view, the execution task that Nextflow assigns to it is an independent job that it runs just like any other batch job. For the Nextflow user, the user experience remains the same while benefiting from all the MMCloud features, such as SpotSurfer, WaveRider, and WaveWatcher.

MMCloud release 2.3.3 introduced a POSIX-compliant, distributed file system that allows Nextflow pipelines to use a high-capacity storage service (such as AWS S3) as a high-performance file system. For complicated pipelines where individual tasks share a file system for writing and reading intermediate data, the performance of the file system has a dramatic impact. 

Running a Nextflow pipeline on MMCloud is straightforward. To demonstrate using AWS EC2 compute instances, we ran [nf-core/rnaseq](https://nf-co.re/rnaseq/3.13.2) on experimental *Eriocheir sinensis* sequence data available from the [European Nucleotide Archive](https://www.ebi.ac.uk/ena/browser/text-search?query=Eriocheir%20sinensis) and the [National Library of Medicine](https://www.ncbi.nlm.nih.gov/datasets/genome/?taxon=95602).

The nf-float plugin requires a configuration file, in which you specify parameters that determine the execution environment. One parameter is the VM creation policy. For this run, we chose "SpotFirst", which means that MMCloud tries to start a spot instance. If, after three attempts, a spot instance does not start, MMCloud starts an on-demand instance. If AWS reclaims the spot instance that a job is running on, MMCloud moves the job to a new instance and follows the same policy - try for a spot instance first and go to an on-demand instance if not successful.

## RNA-Seq Results

At the end of the successful run, Nextflow displayed the following message, which shows that 170 tasks (jobs) completed with no failures.

`-[nf-core/rnaseq] Pipeline completed successfully -`
`Completed at: 25-Nov-2023 10:42:32`
`Duration    : 6h 14m 7s`
`CPU hours   : 123.3`
`Succeeded   : 170`

Included in the output of nf-core/rnaseq is [MultiQC](https://multiqc.info/), a reporting tool that summarizes (in html format) the statistics and quality metrics obtained from multiple analysis modules and data sets. MultiQC is helpful in checking the results of a pipeline run for errors and anomalies.

Examples of the output from MultiQC for the *Eriocheir sinensis* run are shown in the figures that follow.

![](https://baywatch-api.memverge.com/media/upload/202311/231127200149299985.png)

![](https://baywatch-api.memverge.com/media/upload/202311/231127200212828822.png)


## Conclusion

The Chinese mitten crab is a fascinating bundle of surprises. There is no doubt that the species will continue to receive attention - on menus, in departments of environmental conservation, and as the subject of a range of "Omics" studies (for example, [transcriptomics](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6328174/)). As researchers become aware of the power of MMCloud, many of those Nextflow pipelines will run on MMCloud in AWS, Google Cloud, or AliCloud.

For the residents of Shanghai, however, the annual appearance of the Shanghai hairy crab is always an eagerly anticipated milestone.

### What is application observability and why it is important

Application observability gains insights into the behavior of software applications utilizing the information from logs, metrics and traces. The goals include but not limited to

- identify bottlenecks and optimize performance
- identify issues and troubleshoot problems quickly
- identify resource requirements and determine the most appropriate corresponding infrastructure

### What is Memory Machine Cloud WaveWatcher and how it can help

The Memory Machine Cloud WaveWatcher service builds on its ability to track CPU, memory, network, storage (size, throughput, and IO) used by a ‘job’ (could be an application, a workflow, or a pipeline). Take the below WaveWatcher result of a Sentieon WGS pipeline as an example:

![WaveWatcher for Sentieon WGS Pipeline](https://baywatch-api.memverge.com/media/upload/202311/231120214738149768.png)

We can see how this pipeline can be characterized into different steps, each with corresponding ‘behaviors.’ The resources the steps consume are shown in the table below:
| Step | Function | Observability - Memory Usage | Observability - CPU Usage | Observability - Network | Observability - Storage Usage | Observability - Storage IO | What can / should we do? |
| - | - | - | - | - | - | - | - |
1 | Validate license and download reference and sample data | extremely low | extremely low | extensive | increasing | extensive | Use a less memory and CPU intensive but network and storage optimized instance |
2 | BWA-MEM | continuously growing | extremely high | none | increasing | medium | Use a memory and CPU optimized instance and larger instance
3 | Dedup | dynamically changing | dynamically changing | zero | dynamically changing | dynamically changing | Use a CPU optimized instance and larger instance |
4 | Variant calling, filtering and genotyping | high | extremely high | zero | retain | zero | Use a memory and CPU optimized instance and medium instance

### Go beyond observability - Memory Machine Cloud WaveRider

It is helpful to gain observability but the ultimate question is - what can be improved, and can it be automated? The data from WaveWatcher can then be used by the WaveRider service to automatically right-size resources, keep the application / pipeline / workflow state, and reduce CO2 footprint, hence to

- avoid under-provision which may result application performance bottleneck or even OOM, and
- avoid over-provision which spends unnecessary money and CO2 footprint.

More detailed Sentieon WGS pipeline benchmark can be found [here](https://www.mmcloud.io/blog/sentieon-genomics-tools-boosting-performance-wit-3).

### Three pillars of Memory Machine Cloud - observability (WaveWatcher), right-sizing (WaveRider), and job automation (Float)

Last but not the least, how to automate the whole process with minimum human intervention? Just like the three pillars of observability (logs, metrics and traces), there are three pillars for Memory Machine Cloud - job automation (Float), observability (WaveWatcher), and right-sizing (WaveRider).

- The user submits a job (for example, a bioinformatics analysis pipeline) via Float CLI or GUI.
- Memory Machine Cloud WaveWatcher automatically observes the behavior including the resources needed
- Memory Machine Cloud WaveRider automatically right-sizes / vertical scaling based on WaveWatcher’s observation supporting stateful job

These three pillars make a complete cloud automation solution removing the hassle of integrations among multiple solutions.

### Comparison of Memory Machine Cloud with modern solutions

Below is a summary table of comparison of Memory Machine Cloud with modern solutions, where Memory Machine Cloud provides an all-in-one solution.
| Category | Comparison | MemVerge MMCloud | Grafana | Datadog | Spot by NetApp | YellowDog | AWS CloudWatch | AWS Compute Optimizer | AWS Batch |
|:----------:|:---------------------------------------------------------------:|:------------------:|:----------:|:----------:|:--------------:|:---------------:|:--------------:|:---------------------:|:----------:|
| Feature | Observability | Yes (WaveWatcher) | Yes | Yes | Yes | Yes | Yes | Yes | No |
| Feature | Automatic right-sizing stateless application | Yes (WaveRider) | No | No | Yes | Yes | No | No | No |
| Feature | Automatic right-sizing stateful, non-fault-tolerant application | Yes (WaveRider) | No | No | No | No | No | No | No |
| Feature | Job automation | Yes (Float) | No | No | No | Yes | No | No | Yes |
| Deployment | On-prem version | Yes | Yes | No | No | Yes | No | No | No |
| Pricing | Pricing | All-in-one license | By license | By service | By license | Basic + add-ons | By service | Opt-in | By service |

\*note - more detailed comparisons with Spot by NetApp can be found [here](https://www.mmcloud.io/blog/memory-machine-cloud-vs-spot-by-netapp)

### Call for action - save your spending and time

Do you need to run stateful, non-fault-tolerant workloads or pipelines on the cloud? Would you like to benefit from automatic right-sizing and vertical scaling and reduce both cloud cost and job running hours without worrying about integrations among multiple solutions?  [Please try Memory Machine Cloud for free](https://www.mmcloud.io/customer/register).

High-performance computing (HPC) traditionally involves clusters of compute servers deployed in dedicated facilities. In recent years, demands for scale and performance have pushed HPC from on-premises to the cloud - a transition that offers new opportunities as well as significant hurdles.

On the positive side, the cloud offers virtually unlimited on-demand computing and storage resources because cloud platforms provide elastic capacity that can grow (and shrink) dynamically when resource demands change.

There are economic benefits to moving to the cloud, although not always. A stable workload that runs 24x7x365 for several years may be cheaper to run on dedicated hardware (cloud service providers have responded to this case by offering bare-metal instances). The pay-as-you-go model eliminates upfront capital expenses and avoids the costs associated with unused compute cycles. The range of compute instance types available in the cloud allows users to pay only for what they need, when they need it.

However, running applications in public clouds introduces new challenges. Each cloud service provider has an extensive catalog of services, and each service has a wide array of features. Without the knowledge required to configure and optimize cloud services, the researcher may encounter unpredictable or unsatisfactory application performance, unexpected costs, or unnecessary security risks.

This is where Memverge’s Memory Machine Cloud (MMCloud) can help. MMCloud is an easy-to-use cloud automation platform designed for scientific computing and data science. MMCloud simplifies the deployment of data-intensive pipelines and interactive computing applications in the cloud through automation and containerization.

Containers are the key technology that enables application mobility and automates the deployment of applications, at scale, in the cloud. The container includes all the software dependencies and runtime libraries needed to run the application in a reproducible way. But container deployment requires a cloud orchestration platform to schedule, instantiate, and manage containers. MMCloud provides an intuitive GUI (as well as a CLI) that allows the non-cloud specialist to deploy complex workflows as containerized applications in the cloud.

There are many reasons why scientists rely on MMCloud to enhance their research capabilities. Here are the top five reasons why you should choose MMCloud as the platform to run your cloud workloads.

**1. Seamless Expansion of High-performance Computing to the Cloud**

Hybrid cloud (a combination of on-premises and cloud resources) is a popular architecture for HPC because of its flexibility and scalability. When demand for computing resources exceeds what is available locally (on-premises), applications can “burst” to the cloud.

MMCloud effectively augments the local HPC environment with public cloud by enabling the user to launch jobs in the cloud using the same familiar HPC job managers such as SLURM and PBS. This way, scientists can operate in an efficient hybrid environment. Containerization and automation are a core feature of MMCloud. MMCloud automates deployment of containerized applications into the clouds, and it automatically deletes cloud resources after a job completes, removing the risk of unattended resources adding to the monthly cloud bill. Interactive sessions can be hibernated and restarted when needed again.

![Memory Machine Cloud HPC Bridge extends HPC into the cloud](https://baywatch-api.memverge.com/media/upload/202312/231207200802821427.jpeg)


Figure 1 MMCloud expands HPC to clouds

**2. Insightful Monitoring of Cloud Workloads**

Some users are reluctant to move workloads to the cloud because of a concern that they will have limited insight into, and control over, how their applications run in the cloud. Will the job complete in a shorter or longer time? Are the resources over- or under-provisioned?

Visibility into workload performance and resource utilization is critical when operating in the cloud. Granular monitoring of resource usage and costs can identify opportunities for optimization through rightsizing, shutting down idle resources or shifting workloads based on the current prices of compute instances. While cloud service providers do offer monitoring tools, these tools are not always easy to use and often come with additional costs.

MMCloud provides a comprehensive solution with advanced monitoring and metering capabilities, ensuring that scientists have a detailed understanding how their workloads run. CostReporter gives you a summary how much you spend on the jobs running in the cloud. WaveWatcher makes it easy to see the real-time consumption of resources including CPU, memory, storage, and network transfer speeds. For example, an application may exceed its storage limits or crash because of insufficient resources. Issues like these can be identified using MMCloud’s graphical interface, making troubleshooting much more straightforward.

MMCloud generates event and performance logs at both the job and host (worker node) level. Logs can be viewed in real-time to track the progress of a job as it executes, or downloaded after a job completes to identify issues or errors. The monitoring features enable researchers to fine-tune their environments for maximum efficiency.

![MMCloud WaveWatcher provides insights to cloud workloads](https://baywatch-api.memverge.com/media/upload/202311/231121040115584000.png)

Figure 2 MMCloud WaveWatcher provides insights to cloud workloads

**3. Reliable Spot Instances**

Almost all service providers offer spot Instances as a way for users to purchase excess computing capacity at a significantly lower price than on-demand instances (discounts can be as much as 90%). The trade-off is unpredictable availability. The cloud service provider can reclaim these instances with nominal warning (usually two minutes or less). Spot instances are ideal for stateless applications and can be used in stateful, batch-oriented pipelines if the workflow manager can requeue any failed jobs, although the total cost and wall clock time become unpredictable.

MMCloud's SpotSurfer feature allows stateful applications to always run to completion by "surfing" to a new VM instance if the underlying Spot Instance is reclaimed. Customers can realize the cost savings of spot instances without the risk of job interruptions. The technology that makes SpotSurfer possible is AppCapsule. AppCapsule creates a snapshot of the in-memory state as well as file system data. AppCapsule is triggered automatically when the cloud service provider signals that it is reclaiming the spot instance. Job execution pauses and then resumes on a new spot instance.

For interactive sessions, MMCloud provides a reverse proxy to ensure that each interactive session appears to clients as an IP address that does not change even if the session moves to a different virtual machine with a different IP address.
![MMCloud SpotSurfer protects workloads on spot instances](https://baywatch-api.memverge.com/media/upload/202311/231121040319580845.png)

Figure 3 MMCloud SpotSurfer protects workloads on spot instances

**4. Cost Control**

Cost management is a major concern for researchers, many of whom are grant-funded with strict budget constraints. While cloud services can be cost-effective, they can also lead to unpredictable expenses if not managed carefully. Institutions must develop new financial governance models to control cloud spending, ensuring that costs align with research budgets. Managing costs at cloud scale demands continuous monitoring of cloud resource usage to guard against runaway expenses. MMCloud’s Job Cost Reporting tools allow for precise budgeting, and the SurfZone feature provides real-time enforcement of limits on cloud spending.

Lifecycle management of cloud-based workloads can be critical for cost control. MMCloud provides control over of resources, for example, by enforcing automated schedules for job startup and shutdown, and maximum job execution times. This level of control is essential for managing costs in group settings where multiple users may run concurrent experiments. There are actions that individual users can take, for example, canceling jobs, moving jobs to smaller (cheaper) instances, or suspending interactive sessions. These actions can result in saving cost and can be used for avoiding idle charges.

Performance can be critical for cost control too. When jobs run on the spot instances, the slower jobs in data pipelines have a higher chance of getting reclaimed and the time and expense penalty for starting over might be huge. MMCloud’s integration with third-party high-performance file systems has improved the efficiency of file operation and has shown a significant advantage for Nextflow workflows.

WaveRadar, an interactive dashboard created on mmcloud.io is a free FinOps tool for viewing historical spot EC2 prices across all major AWS regions. WaveRadar lets you identify the regions offering the lowest rates for a given instance or instance family.

**5. Rightsizing Cloud Resources**

Compute instances available from cloud service providers range in size from one vCPU and 500 MB memory to 192 vCPUs and 1.5 TB memory (and even more in some specialized instances). Choosing the optimal cloud resources for scientific workloads is a complex task. Sizing for the peak load is safe but inefficient. Sizing for the average load reduces the cost but can lead to unacceptable performance or even out-of-memory (OOM) errors.

MMCloud's WaveRider feature simplifies this process by removing the need to select a single instance for each task. WaveRider provides the capability to migrate running jobs to optimally sized virtual machines based on their real-time resource utilization.

Job migration can be initiated in three ways: manually, driven by policy, or programmatically. At any time, you can manually migrate a job to a specific instance type, or to an instance whose capacities you define. With a rules-based policy, WaveRider monitors the resource utilization of your workload. If the upper threshold for CPU or memory utilization is crossed for a specified interval, the job is migrated to a virtual machine that has more virtual CPUs or more memory. Similarly, if the lower threshold is crossed and utilization remains low for a specified interval, the job is moved to a smaller virtual machine. For jobs with a well-known resource usage profile, job migration can be programmed into the job script so that the job migrates at predetermined stages in the execution.

Rightsizing has a positive effect on the optimal allocation and sharing of cloud resources. The optimization of using cloud resources has a profound impact on environmental sustainability. Users who proactively release underutilized resources allow the cloud service provider to increase server utilization. Fewer servers running at higher utilization lowers operational as well as embedded emissions. In a world increasingly conscious of climate change, optimizing cloud resource usage is a step towards reducing the overall carbon footprint of the IT industry. MMCloud provides a carbon emissions dashboard so that users become aware of the impact of how they run their jobs in the cloud.

![MMCloud Carbon Emission Monitor](https://baywatch-api.memverge.com/media/upload/202311/231121040303736482.png)

Figure 4 MMCloud Carbon Emission Monitor

**Conclusion**

Moving HPC workloads to the cloud, or bursting workloads to the cloud when needed, has benefits in terms of cost, scalability, efficiency, and sustainability. Challenges in usability and cost control in the cloud have prevented users from taking advantage of the cloud or a hybrid cloud architecture.

MMCloud is a powerful, yet easy to use, platform that facilitates the use of cloud computing by the research community. MMCloud is cloud-agnostic and is supported on AWS, Google Cloud, and AliCloud today. A free 30-day, no obligation trial is available. Register at [https://mmcloud.io/customer/register](https://mmcloud.io/customer/register) today.

In the previous <a href="https://www.mmcloud.io/blog/surfzone-using-memory-machine-cloud-to-prevent-clo" rel="noopener noreferrer" target="_blank" style="color: var(--ds-link, #0052CC); background-color: rgb(255, 255, 255);">SurfZone blog</a>, we talked about some real-world examples of cloud budget overruns and their painful consequences. We also mentioned the limitations of existing solutions and how SurfZone can overcome them by providing timely enforcement while saving interim progress.In this blog, we describe the user journeys of both the Cloud Admin and the Cloud User. <h3>Typical user journey</h3>A typical high-level user journey takes 3 steps -<ol><li data-list="ordered">Cloud Admin defines quota group</li><li data-list="ordered">Cloud Admin applies quota group to Cloud User and Cloud Users' submitted jobs will automatically be enforced by the policy of the given quota group.</li><li data-list="ordered">Cloud Admin and Cloud User get notifications when Cloud User's consumption reaches the quota limit or any quota-related actions occur.</li></ol><img src="https://baywatch-api.memverge.com/media/upload/202310/231023012347452899.jpeg"> Now let’s take a closer look at each of the steps along with Memory MachineTM Cloud OpCenter Web GUI.Step 1 - Cloud Admin or Principal Investigator defines a quota group (see Figure 1.1) as follows, and views list of created quota groups (see Figure 1.2).<ol><li data-list="ordered" class="ql-indent-1">What is the quota type you would like to monitor (for example, cost)? What is the quota limit?</li><li data-list="ordered" class="ql-indent-1">What is the usage threshold for notification?</li><li data-list="ordered" class="ql-indent-1">What action should be taken when the usage exceeds the quota? Would it be</li><li data-list="ordered" class="ql-indent-2">Cancel the job, or</li><li data-list="ordered" class="ql-indent-2">Suspend the job (where you can enable auto resume)</li><li data-list="ordered" class="ql-indent-1">What action should be taken when additional quota is applied (for example, auto-resume)?</li></ol><img src="https://baywatch-api.memverge.com/media/upload/202310/231019182008359212.png">Figure 1.1. Cloud Admin adds/edits SurfZone&nbsp;<img src="https://baywatch-api.memverge.com/media/upload/202310/231019182028234291.png">Figure 1.2. Cloud Admin views the list of SurfZone&nbsp;Step 2 - Cloud Admin applies the quota group to a Cloud User (see Figure 2.1 and Figure 2.2) - the Cloud User's jobs are automatically subject to the enforcement mechanisms defined by the quota group (see Figure 2.3).<img src="https://baywatch-api.memverge.com/media/upload/202310/231019182141403006.png">Figure 2.1. Cloud Admin applies SurfZone to Cloud User&nbsp;<img src="https://baywatch-api.memverge.com/media/upload/202310/231019182152185529.png">Figure 2.2. Cloud Admin views the list of Cloud User including quota and usage&nbsp;<img src="https://baywatch-api.memverge.com/media/upload/202310/231019182400294309.png">Figure 2.3. Cloud Admin views the list of Job including their status&nbsp;Step 3 - Cloud Admin (or Principal Investigator) and Cloud User get notifications when user's consumption reaches the quota limit or any quota-related actions occur.<img src="https://baywatch-api.memverge.com/media/upload/202310/231019182429989309.png">Figure 3.1. Cloud Admin gets notifications on users whose quota usage above the configured threshold <img src="https://baywatch-api.memverge.com/media/upload/202310/231019182439198049.png">Figure 3.2. Cloud User views his/her own quota and usage&nbsp;<h3> </h3><h3>What’s next?</h3>What are your thoughts on this feature?If you would like to be on the waitlist to preview this feature, leave your contact information in the comments below.<a href="https://www.mmcloud.io/" rel="noopener noreferrer" target="_blank" style="color: var(--ds-link, #0052CC); background-color: rgb(255, 255, 255);">Memory Machine Cloud by MemVerge</a>

<h3>Who should read this blog? </h3>If you are a Cloud Admin or a Principal Investigator and you are concerned about your team inadvertently exceeding their cloud budget, reading this blog is worth your time (3-minutes).Real-world examples of cloud budget overrunsHere are real-world exampl﻿es of cloud budget overruns...and the painful consequences.In a <a href="http://www.rj-robbins.com/recommended/videos/airi-2021/" rel="noopener noreferrer" target="_blank">presentation</a> on cloud computing in grant-funded research environments, Robert J. Robbins explains how the success of projects can be jeopardized by unexpected (and accidental) expenses. Here are examples.<ol><li data-list="bullet">An expense of $20,000 incurred because a researcher forgot to shut down an AWS instance after completing an analysis - the instance continued to run at $7 an hour for months.</li><li data-list="bullet">An expense of $72,000 incurred because of exponential auto-scaling caused by inserting small tests into product code - the company said they almost went bankrupt.</li></ol> Why aren’t there solutions for this serious problem?Limitations of existing solutionsThere are solutions that monitor cloud cost and can stop running instances, but there are two critical limitations.<ol><li data-list="bullet">Not timely - AWS Budgets, for example, can track cost and usage, but updates typically occur at 8–12 hour intervals. The cloud budget overrun may happen before the next update.</li><li data-list="bullet">Interim progress lost - even if you are alerted when the cloud budget is exceeded and you stop all running instances, interim progress made by unfinished jobs is lost. This means not only the money but the time has been spent and wasted.</li></ol> Is there a single solution to overcome both of these two critical limitations?SurfZone: Memory Machine Cloud's solutionMemory Machine Cloud's SurfZone overcomes both limitations.<ol><li data-list="bullet">Timely - SurfZone provides real-time enforcement for a user's cloud spend (default update interval is one hour but can be configured to a shorter or longer interval).</li><li data-list="bullet">Interim progress saved - When a user reaches the cloud budget limit, SurfZone checkpoints the current application state and puts the application into sleep mode. The application automatically resumes when additional budget is applied.</li></ol> Typical user journeyA typical user journey resembles the following.<ol><li data-list="ordered">Cloud Admin or Principal Investigator defines a quota (budget) group as follows.</li><li data-list="ordered" class="ql-indent-1">What is the quota type you would like to monitor (for example, cost)?</li><li data-list="ordered" class="ql-indent-1">What action should be taken when the usage exceeds the quota (for example, suspend or terminate the job)?</li><li data-list="ordered" class="ql-indent-1">What action should be taken when additional quota is applied (for example, auto-resume)?</li><li data-list="ordered">Cloud Admin or Principal Investigator applies the quota group to a user - the user's jobs are automatically subject to the enforcement mechanisms defined by the quota group.</li><li data-list="ordered">Cloud Admin (or Principal Investigator) and user get notifications when user's spending reaches the quota limit or any quota-related actions occur.</li></ol> What’s next?In a subsequent blog, we’ll take a deeper dive into the user interface that supports SurfZone.What do you think of this feature? Does it work for you or your organization? If you would like to preview this feature, leave your contact information in the comments below.<a href="https://www.mmcloud.io/" rel="noopener noreferrer" target="_blank" style="background-color: var(--ds-surface-raised,white); color: var(--ds-link, #0052CC);"><img src="https://www.mmcloud.io/favicon.ico">Memory Machine Cloud by MemVerge</a>

<h2>Introducing WaveRadar</h2><h3> </h3>There aren't really any great tools for understanding and comparing historical Spot EC2 prices for the 600+ instances available on AWS. To solve this, we created WaveRadar, an interactive dashboard for viewing Spot EC2 prices across all major AWS regions. WaveRadar lets you easily identify the regions offering the lowest $/hour for a given instance or instance family. It can also help you quickly compare the relative price performance $/core and $/memory of different EC2 instances at recent and historical Spot market prices. Use WaveRadar as a free FinOps tool to make more informed decisions around cost optimization, just sign up for a free account via the steps below. To access WaveRadar just navigate to the <a href="https://www.mmcloud.io/aws-spot-pricing" rel="noopener noreferrer" target="_blank" style="background-color: rgb(255, 255, 255);">dashboard_link</a>. <h3>Screenshots of WaveRadar</h3>region optimizer tab<img src="https://hackmd.io/_uploads/HysT_bWkT.png"> visualize/download data tab<img src="https://hackmd.io/_uploads/rJQxY-ZyT.png"><h3> </h3><h3> </h3><h3>Example use cases</h3>Filter/sort the tabulator in the region optimizer tab to find the right instance and region:<ol><li data-list="bullet">Finding the cheapest region for t3.medium spot instance on Sep 12 (3x difference) : <code>lowest recent spot price ($0.0125) at us-east-2b / highest recent spot price ($0.0404) at il-central-1a</code></li><li data-list="bullet">Finding the cheapest region for t3.medium on-demand instance on Sep 12 (1.5x difference) : <code>lowest on-demand price ($0.0416) at us-east-2b / highest on-demand price ($0.0672) at sa-east-1c</code></li><li data-list="bullet">Find a cheap instance with 4 CPUs and 16 GiB memory in us-west-1c on Sep 14 : <code>lowest recent spot price ($0.0480) for t4g.xlarge / highest recent spot price ($0.0662) for t2.xlarge </code></li></ol> Visualize and download spot price history in the visualize/download data tab:<ol><li data-list="bullet" class="ql-indent-1">Access historical spot price data from April to the current date for informed decision-making.</li></ol>

SummaryThe <a href="https://www.sentieon.com/" rel="noopener noreferrer" target="_blank">Sentieon®</a>&nbsp;Genomics tools provide a complete rewrite — stressing computational efficiency, accuracy, and consistency — of popular genomics pipelines (such as GATK, Picard, and MuTect) for calling variants from next-generation sequence data. Consistently, Sentieon Genomics tools show a five- to ten-fold improvement in performance over the equivalent open-source pipelines.A common characteristic of variant calling pipelines is that the resource demands on CPU cores and memory access vary as the pipeline executes. MemVerge’s&nbsp;<a href="https://www.mmcloud.io/" rel="noopener noreferrer" target="_blank">Memory Machine Cloud</a>&nbsp;includes a checkpoint/restore feature (called WaveRider) that can automatically migrate a running container, without losing execution state, to a virtual machine of different capacity as soon as a change in resource requirements is detected.When Sentieon Genomics tools are used in conjunction with WaveRider, performance (measured by wall clock time) improves even further because processes are never starved for resources. The costs associated with pipeline execution also decrease because virtual machines are never overprovisioned. Depending on how EC2 On-demand instance prices scale with the number of vCPUs and memory capacity, it is possible to optimize cost and performance simultaneously by running a larger virtual machine for a shorter period when CPU and/or memory utilization are high, and a smaller virtual machine for a longer period when CPU and/or memory utilization are low.The Sentieon-developed whole genome sequencing (WGS) pipeline benchmark simulates a typical bioinformatic analysis pipeline, starting with loading raw data (in the form of fastq files) and ending with output in the form of vcf files. The <a href="https://www.mmcloud.io/blog/sentieon-genomics-tools-boosting-performance-wit-3" rel="noopener noreferrer" target="_blank">complete report</a> compares the performance of the WGS pipeline benchmark executed without using MMCloud to the performance when used with MMCloud.The tests show that, compared to a baseline measured by running the WGS pipeline benchmark on a single AWS EC2 instance, the combination of Sentieon Genomics tools and Memory Machine Cloud (with WaveRider) demonstrates a 40% decrease in wall clock time and a 34% reduction in cost.

<h2>Introduction 📣</h2>Are you a bioinformatics researcher looking to supercharge your Nextflow workflows? 🚀 Do you want to explore the future of cloud computing while contributing to a groundbreaking initiative? 🌐 If so, we have some exciting news for you! MemVerge is thrilled to announce the MMCloud NF-Surfer Program 🏄, an initiative designed to invite Nextflow/nf-core researchers to experience the unparalleled capabilities of our Memory Machine Cloud (MMCloud). 🎉<a href="https://forms.gle/7doKyVg8CPhR17C48" rel="noopener noreferrer" target="_blank"><img src="https://baywatch-api.memverge.com/media/upload/202309/230905204224567226.png"></a> <h2>What is MMCloud? ☁️</h2>MMCloud is MemVerge's state-of-the-art cloud computing platform that leverages the power of Memory-Converged Infrastructure (MCI) to offer high-speed, scalable, and reliable computing resources. 💪 It is designed to handle complex workflows and data-intensive tasks with ease on multiple clouds, making it an ideal choice for nf-core researchers working on Nextflow workflows. 🛠️<h2>Running Nextflow with MMCloud as the Computing Environment on AWS:</h2><iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://www.youtube.com/embed/qlNVnCZAkFk?si=HNPUewqQ-2eXhluE"></iframe> <h2>For any queries or additional information, feel free to reach out to us in the <code class="ql-size-small" style="background-color: rgb(240, 240, 240);">#platform-memverge</code> channel on the Nextflow Tower Slack workspace.</h2> <h2>🔄Why Participate? 🌟</h2><ol><li data-list="bullet">Accelerate Your Research: MMCloud's cloud automation removes complexity and significantly speeds up your Nextflow workflows. ⚡</li><li data-list="bullet">Contribute to Innovation: Your feedback will be instrumental in shaping the future of cloud computing for nf-core researchers. 🌐</li><li data-list="bullet">Financial Incentives: Receive free AWS credits for your projects as a token of our appreciation for your participation. 💵</li></ol><h2> </h2><h2>Conclusion 🎬</h2>The MMCloud NF-Surfer Program is a unique opportunity for Nextflow/nf-core researchers to experience the future of cloud computing while contributing to a meaningful initiative. 🌈 Space is limited, So why wait? Apply today and ride the wave of innovation with MMCloud! 🏄‍♀️🏄‍♂️Let's surf the cloud together! 🌊🏄‍♀️🏄‍♂️

<h2>MemVerge nf-float plugin is now included in the Nextflow plugins index</h2>MemVerge is pleased to announce our Nextflow plugin is now included in the Nextflow plugins index repository&nbsp;<a href="https://github.com/MemVerge/nf-float" rel="noopener noreferrer" target="_blank" style="color: inherit;">on GitHub</a>. This means that any process in a Nextflow pipeline can call MemVerge Memory Machine Cloud as an executor. Nextflow will automatically search the Plugins index and download the latest version of nf-float.Nextflow currently supports a broad range of executors. Processes can be scheduled to run on the local host, on an HPC cluster, on Kubernetes, or on a cloud platform such as AWS or GCP. Memory Machine Cloud adds several features that provide significant value to Nextflow users in the cloud.<a href="https://www.mmcloud.io/blog/unveiling-the-mmcloud-nf-surfer-program-a-golden-o" rel="noopener noreferrer" target="_blank"><img src="/img/case_studies/nextflow_04.png" alt="Nextflow"></a><h2>nf-float unlocks the added value of Memory Machine Cloud</h2>Memory Machine Cloud is software that is deployed on virtual machines in any cloud including AWS, GCP and AliCloud today, with more coming soon.<h2>OpCenter</h2>The heart of Memory Machine Cloud is the OpCenter that schedules jobs and instantiates the appropriate resources to run containerized applications. The OpCenter continually monitors the resources consumed by jobs<h2>WaveRider Continuous Optimization Service</h2>Based on user-supplied policy, moves running jobs to optimally-sized virtual machines, all without losing state or interim progress. This WaveRider service enables users to tune cost and wall clock time during job runtime.<h2>SpotSurfer Checkpointing and Recovery Service</h2>Spot instances are a way for cloud service providers to monetize idle compute capacity. A Spot instance may be offered at a discount of up to 90% off the price of an On-demand instance. There is a catch — a Spot instance may be reclaimed by the cloud service providers with a typical warning of only two minutes. Memory Machine Cloud includes the SpotSurfer checkpoint and restore service that allows a running job to move seamlessly from a Spot instance flagged for reclamation to a new Spot or on-demand instance. This feature, called SpotSurfer, allows users to reduce cloud costs significantly without increasing wall clock time.<a href="https://www.mmcloud.io/case-studies/tgen" rel="noopener noreferrer" target="_blank">See how TGen used Nextflow &amp; SpotSurfer</a><h2>WaveWatcher</h2>When the OpCenter launches a worker node, the OpCenter opens a communication channel to a resource monitor inside the container. Real-time metrics are available as a CSV file or can be viewed graphically using the OpCenter web interface. This feature, called WaveWatcher, provides Nextflow users with a detailed view of how executors are consuming compute and storage resources, an insight not available otherwise.What does this mean for Nextflow users? Memory Machine Cloud is a feature-rich executor, easily deployed with a few lines in the Nextflow configuration file. The benefits include cost savings, improved time-to-results, and deep insight into resource utilization at the container level.Read&nbsp;<a href="https://docs.memverge.com/mmce/2.1/howto/nextflow/pdf/mmce_2_1_howto_nextflow.pdf" rel="noopener noreferrer" target="_blank" style="color: inherit;">Using Nextflow with Memory Machine Cloud&nbsp;</a>for detailed instructions on how to deploy Memory Machine Cloud with its Nextflow plugin.