This topic is resolved

Share
Viewing 2 reply threads
  • Author
    Posts
    • #1161326
      Bargavi
      Participant
      • Topics: 2
      • Replies: 0
      Member Points: 125
      Rank: Level 2

      I would like to launch multiple Amazon EC2 spot instances (fleet?) using a custom AMI (docker?) for performing a deep-learning training task. I would like all the instances to share a common set of files for the purposes of training the model.

      The idea here is not to lose training history and keep a backup in EBS (network drive?) AWS Certified when the spot instance is terminated by AWS due to pricing-limit/demand. The task state can be updated in a file and then resumed when instances are available.

      Is it possible to launch all instances and let them work cooperatively to complete the training task? What kind of setup could accomplish this?

      1+
      avatar
    • #1161342
      Michael Pietroforte
      Keymaster
      • Topics: 170
      • Replies: 640
      Post count: 1909
      Member Points: 25,469
      Author of the year 2018
      Rank: Level 4

      It depends on the size of the project. With AWS ClouldFormation you can coordinate the provisioning of multiple AWS resources.

      I am curious. What deep learning task is this? Which software are using?

      0
    • #1161352
      Swapnil Kambli
      Moderator
      • Topics: 0
      • Replies: 60
      Post count: 49
      Member Points: 4,869
      Rank: Level 3

      Hi Bargavi,

      Spot Fleet would do the trick for you to launch and maintain the number of machines running.

      Here is the step by step tutorial for the similar requirement that you have.

      https://aws.amazon.com/blogs/machine-learning/train-deep-learning-models-on-gpus-using-amazon-ec2-spot-instances/

      Kindly reply back if you need to tweak or customize the solution.

      1+
      avatar
Viewing 2 reply threads
  • You must be logged in to reply to this topic.
© 4sysops 2006 - 2020

CONTACT US

Please ask IT administration questions in the forums. Any other messages are welcome.

Sending

Log in with your credentials

or    

Forgot your details?

Create Account