Help! Transfer data between EC2 and S3

This forum is for issues related to parallel processing
and OpenSees using the new interpreters OpenSeesSP and OpenSeesMP
Post Reply
blzopensees
Posts: 2
Joined: Tue Dec 06, 2016 6:00 am

Help! Transfer data between EC2 and S3

Post by blzopensees » Thu Dec 22, 2016 12:11 am

I am trying to transfer earthquake records from S3 to EC2. I tryed to use Starcluster to start the instances with IAM role with reference to the following page:
http://star.mit.edu/cluster/mlarchives/2314.html
However, when I tried to copy the file 'main.tcl' from my bucket 'wodecunchutong' to directory 'root' on the instances, an error occurred as follows:

C:\Users\asus>starcluster put mycluster s3://wodecunchutong/main.tcl /root
StarCluster - (http://star.mit.edu/cluster) (v. 0.95.6)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

!!! ERROR - Local file or directory does not exist:
!!! ERROR - s3://wodecunchutong/main.tcl

I also tried to use AWS CLI with reference to the following page:
http://stackoverflow.com/questions/1591 ... nd-aws-ec2
, but still failed:

C:\Users\asus>aws s3 cp s3://wodecunchutong/main.tcl /root --recursive
C:\Users\asus>starcluster sshmaster mycluster
StarCluster - (http://star.mit.edu/cluster) (v. 0.95.6)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

>>> Starting Pure-Python SSH shell...
Line-buffered terminal emulation. Press F6 or ^Z to send EOF.

eval $(resize)
root@master:~# eval $(resize)
7[r[999;999H[6n
resize: Time out occurred
root@master:~# ls /root
ls /root

That is to say, nothing was copied to the /root directory on the EC2 instance.

Can anyone tell me how to copy files from EC2 to S3 bucket? I am new to Amazon EC2 and S3.Please give me some advice.
Thanks!

fmk
Site Admin
Posts: 5860
Joined: Fri Jun 11, 2004 2:33 pm
Location: UC Berkeley
Contact:

Re: Help! Transfer data between EC2 and S3

Post by fmk » Mon Jan 30, 2017 9:41 pm

it obviosly does not like the directory: s3://wodecunchutong/main.tcl

you could try just cd'ing into the directory and then issing the command:
starcluster put mycluster main.tcl /root

cathyfernandez
Posts: 2
Joined: Thu Nov 21, 2019 9:39 am

Re: Help! Transfer data between EC2 and S3

Post by cathyfernandez » Thu Nov 21, 2019 9:44 am

Hi Its glad to be here even though I have 2+ experience in server side I need a clear Idea but I have come up with some suggestion below

Run aws configure and enter above credentials. For region, use 'us-east-1'.
Check that you have access to s3 bucket with aws s3 ls .
Copy entire directory. I recommend using sync rather than cp . Note this part tends to hang sometimes, so just ctrl+c and run it again. By sync , the process will resume.

for more visit https://hopetutors.com/course/aws-train ... i-chennai/

Post Reply