amazon s3 - Opscenter backup failure - S3 500 InternalError -

-

the backup service using opscenter failing below aws s3 error - 500 internalerror. failing on 3-4 nodes different files error code. aws s3 documentation recommends retry operation (we encountered internal error. please try again). can see file name , error once in agent log , s3 log, means there no retry done after error. there way enable retry s3 500 error code (internalerror) in opscenter? suggestion on how fix error?

 
  
error while sending opsagent.backups.mytable.sstable@1d6cb0c5   org.jclouds.http.httpresponseexception: request: head https://my-backups.s3.amazonaws.com/snapshots/a3b72e7b-bd70-4f9e-aa2a-
  
cf6c2a5ff336/sstables/1458057843-my_ks-mytable-ka-57417-index.db.gz http/1.1 failed response: http/1.1 500 internal server error         @ org.jclouds.aws.handlers.parseawserrorfromxmlcontent.handleerror(parseawserrorfromxmlcontent.java:63)         @ org.jclouds.http.handlers.delegatingerrorhandler.handleerror(delegatingerrorhandler.java:67)         @ org.jclouds.http.internal.basehttpcommandexecutorservice.shouldcontinue(basehttpcommandexecutorservice.java:135)         @ org.jclouds.http.internal.basehttpcommandexecutorservice.invoke(basehttpcommandexecutorservice.java:105)         @ org.jclouds.rest.internal.invokesynctoasynchttpmethod.invoke(invokesynctoasynchttpmethod.java:128)         @ org.jclouds.rest.internal.invokesynctoasynchttpmethod.apply(invokesynctoasynchttpmethod.java:94)         @ org.jclouds.rest.internal.invokesynctoasynchttpmethod.apply(invokesynctoasynchttpmethod.java:55)         @ org.jclouds.rest.internal.delegatestoinvocationfunction.handle(delegatestoinvocationfunction.java:156)         @ org.jclouds.rest.internal.delegatestoinvocationfunction.invoke(delegatestoinvocationfunction.java:123)         @ com.sun.proxy.$proxy51.objectexists(unknown source)         @ org.jclouds.s3.blobstore.s3blobstore.blobexists(s3blobstore.java:175)         @ org.jclouds.blobstore2$blob_exists_qmark_.invoke(blobstore2.clj:238)         @ opsagent.backups.destinations$create_blob.invoke(destinations.clj:48)         @ opsagent.backups.destinations$fn__12755.invoke(destinations.clj:185)         @ opsagent.backups.destinations$fn__12385$g__12378__12396.invoke(destinations.clj:25)         @ opsagent.backups.staging$start_staging_bang_$fn__12925$state_machine__5264__auto____12926$fn__12931$fn__12962.invoke(staging.clj:61)         @ opsagent.backups.staging$start_staging_bang_$fn__12925$state_machine__5264__auto____12926$fn__12931.invoke(staging.clj:59)         @ opsagent.backups.staging$start_staging_bang_$fn__12925$state_machine__5264__auto____12926.invoke(staging.clj:56)         @ clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)         @ clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)         @ clojure.core.async.impl.ioc_macros$take_bang_$fn__5280.invoke(ioc_macros.clj:953)         @ clojure.core.async.impl.channels.manytomanychannel$fn__1785.invoke(channels.clj:102)         @ clojure.lang.afn.run(afn.java:24)         @ java.util.concurrent.threadpoolexecutor.runworker(unknown source)         @ java.util.concurrent.threadpoolexecutor$worker.run(unknown source)         @ java.lang.thread.run(unknown source)
  1

details:
opscenter 5.2.4
dse 4.8.2
data size per node ~130gb x 3 nodes x 3 dc
compression , s3 server side encryption enabled
jvm_opts="$jvm_opts -xmx512m -djclouds.mpu.parts.magnitude=100000 -djclouds.mpu.parts.size=32000000"

there issue in opscenter 5.2.4 placed blob_exists code outside of our retry loop. has been fixed in opscenter 6.0.1.


Comments

Post a Comment

Popular posts from this blog

Combining PHP Registration and Login into one class with multiple functions in one PHP file -

-
i'm trying combine registration, activation , login scripts php website backend script front end developer can pass variables different forms to. my question whether appropriate approach this. don't want have lot of php files different pieces of application developing. far have written following 2 functions login, register , activate user front end developer can call: <?php /** * created phpstorm. * user: karl * date: 26/07/2016 * time: 02:25 */ class users { function register_user($email, $password, $user_name) { $server_name = "localhost"; $u_name = "root"; $db_password = "root"; $db_name = "betamath_graspe"; //email notification variable $from_address="info@slack.com"; //registration form $msg_reg_user='username taken. please choose different username'; $msg_reg_email='email registered'; $msg_reg_active=...
Read more

magento2 - Magento 2 admin grid add filter to collection -

-
i created grid in admin xml ui components. need filter collection via url parameter , dont know how achieve that. tried inject requestinterface collection, filter didnt work. di.xml <virtualtype name="slidelistingdataprovider" type="magento\framework\view\element\uicomponent\dataprovider\dataprovider"> <arguments> <argument name="collection" xsi:type="object" shared="false">xxx\xxx\model\resourcemodel\grid\slide\collection</argument> <argument name="filterpool" xsi:type="object" shared="false">slidelistingfilterpool</argument> <!-- define new object filters --> </arguments> </virtualtype> <virtualtype name="slidelistingfilterpool" type="magento\framework\view\element\uicomponent\dataprovider\filterpool"> <arguments> <argument name="applier...
Read more

Android volley - avoid multiple requests of the same kind to the server? -

-
i using volley upload list of images server. happens within service. once image uploaded remove list. problem arises when internet connection breaks. when internet connection breaks , comes again thinking of adding images yet uploaded request queue. this can result in more 1 copy of same image saved on server request same image might have gone through before getting rseponse. how can address scenario? i'm not quite sure why you're using service schedule requests volley automatically runs it's request in seperate thread. nevertheless can listen both request results, successful , unsuccessful adding listeners. you can remove image list schedule request using volley. if succeeds can carry on desired, if fails can add again list. public static request getimageuploadrequest(final string image) { response.listener<t> responselistener = new response.listener<t>() { @override public void onresponse(t response...
Read more