Using Cost Allocation Report on Amazon Web Services (AWS)

-

AWS offers web services like compute instances. Lately, I have been using one cc2.8xlarge instance for 3 hours on a weekly basis to give training sessions. My 14 students connect to orion.cloud.raytrek.com (a canonical name to my AWS instance) during every training session.The instance has one additional 300 GiB EBS volume attached to it so that my students keep their data for the whole duration of the training program.

On AWS, I can tag anything I use: EC2 instances, EC2 EBS volumes, S3 buckets, and so on. A tag is a key and a value (key=value), for example Project=Ray-Cloud-Browser-public-demo. On AWS, it's possible to activate a feature called Cost Allocation Report. This feature deposits detailed usage reports in one S3 bucket that I own. These reports include costs.

I tagged my Cost Allocation Report S3 bucket with Project=Billing to get a grasp on on much it costs to use the Cost Allocation Report feature. The cost of getting my Cost Allocation Report reports is only $ 0.02.

I wrote a Ruby script that generates pivot tables for my projects. Below is my Cost Allocation Report tables with some confidential information redacted. Things under Project=not-classified are things that were not tagged.

Pivot table for Cost Allocation Report on AWS
File from AWS S3: #####################-aws-cost-allocation-2013-02.csv
Project=ray-in-cloud-cc2.8xlarge-CLI
Product CodeUsage TypeUnitsUsage QuantityTotal Cost ($)
AWSDataTransferDataTransfer-Regional-Bytes GB 2.8E-70.0
AWSDataTransferDataTransfer-In-Bytes GB 0.000060330.0
AmazonEC2SpotUsage:cc2.8xlarge instance-hours 10.000000002.7
AWSDataTransferDataTransfer-Out-Bytes GB 0.000020290.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.000044145.0e-06



Total=2.700005
Project=Ray-Cloud-Browser-##############
Product CodeUsage TypeUnitsUsage QuantityTotal Cost ($)
AWSDataTransferDataTransfer-Regional-Bytes GB 0.000047656.0e-06
AWSDataTransferDataTransfer-In-Bytes GB 0.000002860.0
AWSDataTransferDataTransfer-In-Bytes GB 8.656911690.0
AmazonEC2SpotUsage:t1.micro instance-hours 338.751381221.017348
AmazonEC2EBS:VolumeUsage GB-months 36.334022093.633367
AmazonEC2EBS:VolumeIOUsage I/O requests 3861215.847887990.385104
AWSDataTransferDataTransfer-Out-Bytes GB 0.000002300.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.181801450.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.000005001.0e-06
AWSDataTransferDataTransfer-Out-Bytes GB 0.395612710.047268



Total=5.083094
Project=formation-#############-bioinformatique-hiver-2013
Product CodeUsage TypeUnitsUsage QuantityTotal Cost ($)
AWSDataTransferDataTransfer-Regional-Bytes GB 0.000452776.0e-05
AWSDataTransferDataTransfer-In-Bytes GB 3.909986890.0
AmazonEC2BoxUsage:cc2.8xlarge instance-hours 10.0000000024.0
AmazonEC2EBS:VolumeUsage GB-months 145.4776285614.547622
AmazonEC2EBS:VolumeIOUsage I/O requests 267824.572144110.026712
AWSDataTransferDataTransfer-Out-Bytes GB 0.102163410.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.222314740.026562



Total=38.600956000000004
Project=not-classified
Product CodeUsage TypeUnitsUsage QuantityTotal Cost ($)
AWSDataTransferDataTransfer-Regional-Bytes GB 0.074354980.009904
AWSDataTransferDataTransfer-Regional-Bytes GB 0.000006451.0e-06
AWSDataTransferDataTransfer-In-Bytes GB 0.001651900.0
AWSDataTransferDataTransfer-In-Bytes GB 0.043491390.0
AWSDataTransferDataTransfer-In-Bytes GB 0.042701760.0
AWSDataTransferDataTransfer-In-Bytes GB 0.043251740.0
AWSDataTransferDataTransfer-In-Bytes GB 5.474990610.0
AWSDataTransferDataTransfer-In-Bytes GB 0.000001970.0
AWSDataTransferDataTransfer-In-Bytes GB 0.000185850.0
AmazonS3Requests-Tier1 HTTP requests 26.426229510.001066
AmazonEC2BoxUsage:cc2.8xlarge instance-hours 1.000000002.4
AmazonEC2SpotUsage:t1.micro instance-hours 1.026519340.003083
AmazonEC2SpotUsage:t1.micro instance-hours 250.470718230.752221
AmazonEC2EBS:VolumeUsage GB-months 28.044400062.804413
AmazonEC2DataProcessing-Bytes
0.001464110.01
AmazonSNSRequests-Tier1 HTTP requests 459.000000000.0
AmazonEC2EBS:VolumeIOUsage I/O requests 3051481.508183820.304344
AmazonEC2SpotUsage:cr1.8xlarge instance-hours 9.000000003.09
AmazonEC2LoadBalancerUsage
1.000000000.03
AmazonEC2SpotUsage:cc2.8xlarge instance-hours 10.000000002.7
AWSDataTransferDataTransfer-Out-Bytes GB 0.000252600.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.002935500.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.001021180.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.000263750.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.147749280.0
AWSDataTransferDataTransfer-Out-Bytes GB 4.3E-70.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.000405910.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.000549686.6e-05
AWSDataTransferDataTransfer-Out-Bytes GB 0.006387850.000763
AWSDataTransferDataTransfer-Out-Bytes GB 0.002222160.000266
AWSDataTransferDataTransfer-Out-Bytes GB 0.000573946.9e-05
AWSDataTransferDataTransfer-Out-Bytes GB 0.321512800.038415
AWSDataTransferDataTransfer-Out-Bytes GB 9.4E-70.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.000883290.000106
AmazonEC2SpotUsage:cr1.8xlarge instance-hours 3.000000001.03
AmazonEC2SpotUsage:cc2.8xlarge instance-hours 1.000000000.27



Total=13.444717
Project=Ray-Cloud-Browser-public-demo
Product CodeUsage TypeUnitsUsage QuantityTotal Cost ($)
AWSDataTransferDataTransfer-Regional-Bytes GB 0.000213352.8e-05
AWSDataTransferDataTransfer-In-Bytes GB 0.000020070.0
AWSDataTransferDataTransfer-In-Bytes GB 0.226134370.0
AmazonEC2SpotUsage:t1.micro instance-hours 338.751381221.017348
AmazonEC2EBS:VolumeUsage GB-months 36.334022093.633367
AmazonEC2EBS:VolumeIOUsage I/O requests 3726756.784679400.371693
AWSDataTransferDataTransfer-Out-Bytes GB 0.000016130.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.560928340.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.000035104.0e-06
AWSDataTransferDataTransfer-Out-Bytes GB 1.220619400.145841



Total=5.168280999999999
Project=ray-in-cloud-cc2.8xlarge
Product CodeUsage TypeUnitsUsage QuantityTotal Cost ($)
AWSDataTransferDataTransfer-In-Bytes GB 0.043436850.0
AmazonEC2SpotUsage:cc2.8xlarge instance-hours 10.000000002.7
AWSDataTransferDataTransfer-Out-Bytes GB 0.002327370.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.005064520.000605



Total=2.7006050000000004
Project=Billing
Product CodeUsage TypeUnitsUsage QuantityTotal Cost ($)
AWSDataTransferDataTransfer-In-Bytes GB 0.003016820.0
AmazonS3Requests-Tier1 HTTP requests 221.573770490.008934
AmazonS3Requests-Tier2 HTTP requests 362.000000000.01
AmazonS3TimedStorage-ByteHrs GB 0.000048820.01
AWSDataTransferDataTransfer-Out-Bytes GB 0.000112060.0
AWSDataTransferDataTransfer-Out-Bytes GB 0.000243862.9e-05



Total=0.028963000000000003
Project=Ray-TestSuite
Product CodeUsage TypeUnitsUsage QuantityTotal Cost ($)
AmazonEC2EBS:VolumeUsage GB-months 0.012308270.001231
AmazonEC2EBS:VolumeIOUsage I/O requests 21529.287104680.002147



Total=0.003378
Project=###############-instance-testing
Product CodeUsage TypeUnitsUsage QuantityTotal Cost ($)
AmazonEC2BoxUsage:hs1.8xlarge instance-hours 1.000000004.6



Total=4.6

People at Amazon.com, Inc. always say that they are obsessed by theirs customers. I am a happy customer of Amazon Web Services, Inc. (AWS), and I can confirm that AWS is really easy for the customer for many reasons, like the Cost Allocation Report.

Cost Allocation Report is really a feature for the customer that allows a better understanding of costs in the cloud. AWS could have charged a lot for that kind of feature -- banks charge their customers a lot for getting account statements from 3 years ago.


p.s.: I have no financial or commercial links with AWS, I am really just one happy AWS customer. I really think that AWS is giving me a great service for the money I give them. It's a win-win situation.

Comments

Beyond said…
Amazon S3 provides a secure, reliable, scalable and inexpensive, "pay only for what you use" storage service on the internet. So I like to use it very well.
Bucket Tagging is very helpful to segment our s3 bill according to the tag added on the buckets.

Regards
Ronak Jain
Bucket Explorer
Bucket Explorer is graphical user interface (GUI) to manage data on Amazon S3 in an efficient and user friendly way.
Friday, March 8, 2013 at 1:10:00 AM EST
atg2taa said…
This comment has been removed by the author.
Wednesday, July 31, 2013 at 6:06:00 PM EDT
sebhtml said…
atg2taa wrote:
> Hi Sébastien, I'm interested in
> using AWS for genome assemblies.
> I'm looking at the super large
> memory instances but some of the
> new terminology is hard to
> translate into terms I'm used to.
>
> Could you comment a bit on the
> pricing and time costs of doing,
> say a 50Mb genome using AWS?

You can use the spot market -- it is cheap and you can have access to nice machines.

The cr1.8xlarge has 32 vCPUs, 244 GiB memory and even SSDs !

(see http://aws.amazon.com/ec2/instance-types/instance-details/ )

cr1.8xlarge is around 0.35 $ per hour in the spot market.

>
> Is transferring/reading data from
> EBS just as fast as reading from
> local disk?

EBS is tunable. Regular EBS is probably slower, but you can provision a number of operations per second or use EBS-optimized instances which uses a dedicated network channel for EBS.

Since cr1.8xlarge has SSD disks, you can also use these (but they are ephemeral).

But Ray only need disks to load input data and store results. All computation is in-memory.



>
> Thanks,
Friday, August 2, 2013 at 11:04:00 AM EDT
Post a Comment

Popular posts from this blog

A survey of the burgeoning industry of cloud genomics

-
Image
I like lists. Here is a list of companies in the cloud genomics industry. Table 1: Companies acting in the industry of cloud genomics. The list is in no particular order. Company Products People Founded Link GenoSpace, LLC - GenoSpace platform - John Quackenbush (CEO) 2011 https://www.genospace.com/ DNAnexus, Inc. - DNAnexus Platform - DNAnexus Platform SDK (a.k.a. dx-toolkit) - Serafim Batzoglou (co-founder) - Andreas Sundquist (co-founder and CTO) - a bunch of other famous people in the field 2009 https://dnanexus.com/ Seven Bridges Genomics, Inc. - IGOR - IGOR Python SDK - Igor Bogicevic (CTO) 2009 https://www.sbgenomics.com/ BGI - EasyGenomics - Xing Xu (Senior Product Manager) Introduced in 2012 https://www.easygenomics.com/ Illumina, Inc. - BaseSpace™ - BaseSpace API - Alex Dickinson (SVP of Cloud Genomics) Introduced in 2011 https://basespace.illumina.com/ Era7 Information Technologies SLU - BG7...
Read more

Generating neural machine instructions for multi-head attention

-
  An animal can do training and inference every day of its existence until the day of its death. A forward pass is all you need.   Here is the current model configuration that I use for multi-head attention:         let sequence_length = 6;         let vocab_size = 20;         let n_embd = 64;         let num_heads = 8; You can see the full model here: https://github.com/sebhtml/novigrad/blob/8df6192829edd9cda3a221e12affa770fbda2a1b/src/models/mega_man_attention.rs The struct uses Embedding, MultiHeadAttention, Linear, Softmax: struct MegaManAttentionModel {     input_shape: Vec<usize>,     output_shape: Vec<usize>,     vocab_size: usize,     sequence_length: usize,     embedding: Embedding,     multi_head_attention: MultiHeadAttention, ...
Read more

Adding ZVOL VIRTIO disks to a guest running on a host with the FreeBSD BHYVE hypervisor

-
On atlantis.domain (my FreeBSD BHYVE hypervisor host), I use these commands: Install FreeBSD on a 10G disk. vm create -s 10G guestname   vm install FreeBSD-12.1-RELEASE-amd64-memstick.img vm console guestname Add 8 30GB zvol-based disks vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname You will need to do vm poweroff guestname vm start guestname because the guest does not seem to see the 8 30GB without power cycling. (rebooting won't work) *** If you want the boot disk (the one create by "vm create") to be a zvol, you must edit the template: root@atlantis:~ # zfs snapshot tank/vm-bhyve@2020-06-11-2145 root@atlantis:~ # mount -t zfs tank/vm-bhyve@2020-06-11-2145 /tank/vm-bhyve@2020-06-11-2145 r...
Read more