Script disconnected unexpectedly while long curl #7323

yaniveliash · 2019-02-18T13:33:08Z

Seems like long curl (about 2 minutes)
disconnect my session in the middle.

That isn't consist as sometime it works and sometimes not...

Packer version : 1.3.3
Platform : AWS EBS (Ubuntu)

Debug log:
https://gist.github.com/geostant/16f8ade96ecbdecc0a30e99df6ad7135

yaniveliash · 2019-02-20T09:11:31Z

I tried today to use wget, got the same issue

umerebryx · 2019-09-11T05:31:14Z

Any update on this? I am also facing the same issue if a command runs for longer time packer build process is killed.

yaniveliash · 2019-09-11T06:14:32Z

@umerebryx
I gave up on this a long time ago...

azr · 2019-09-11T09:00:49Z

Hello there; the original gist posted is a 404. But I think this issue could be due to the fact that the communicator will hang up after a while if nothing happens.

yaniveliash · 2019-09-11T09:23:00Z

@azr In my case the curl was against docker domain (getting the installation file)
I think it's responsive enough

azr · 2019-09-11T09:33:09Z

Do you have the configuration to reproduce it ?

umerebryx · 2019-09-11T09:39:11Z

In my case it abruptly fails while the script is running and there is continuous output. Here is where my build stops:

amazon-ebs: zlib 1.0.0
amazon-ebs: installing bundled gems: /usr/lib/ruby/gems/2.5.0 (build_info, cache, doc, extensions, gems, specifications)
amazon-ebs: test-unit 3.2.7
amazon-ebs: power_assert 1.1.1
amazon-ebs: minitest 5.10.3
amazon-ebs: xmlrpc 0.3.0
amazon-ebs: rake 12.3.0
amazon-ebs: net-telnet 0.1.1
amazon-ebs: did_you_mean 1.2.0
amazon-ebs: installing rdoc: /usr/share/ri/2.5.0/system
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [ERROR] Remote command exited without exit status or exit signal.
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [INFO] RPC endpoint: Communicator ended with: 2300218
2019/09/11 12:13:15 [INFO] 100078 bytes written for 'stdout'
2019/09/11 12:13:15 [INFO] 15982 bytes written for 'stderr'
2019/09/11 12:13:15 [INFO] RPC client: Communicator ended with: 2300218
2019/09/11 12:13:15 [INFO] RPC endpoint: Communicator ended with: 2300218
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [INFO] 15982 bytes written for 'stderr'
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [INFO] 100078 bytes written for 'stdout'
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [INFO] RPC client: Communicator ended with: 2300218
2019/09/11 12:13:15 [INFO] (telemetry) ending shell
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
2019/09/11 12:13:57 [INFO] (telemetry) ending amazon-ebs
2019/09/11 12:13:57 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2019/09/11 12:13:57 machine readable: amazon-ebs,error []string{"Script disconnected unexpectedly. If you expected your script to disconnect, i.e. from a restart, you can try adding \"expect_disconnect\": true or \"valid_exit_codes\": [0, 2300218] to the shell provisioner parameters."}
==> Builds finished but no artifacts were created.
2019/09/11 12:13:57 [INFO] (telemetry) Finalizing.

Another thing I observed is the script always fails at 6 mins and ~47 sec which make me think if there is a global timeout set somewhere that is causing this behaviour?

My packer version is 1.4.3 and running it on Mac.

azr · 2019-09-11T10:17:28Z

Do you have easy to reproduce steps ?

It looks like to me that the command crashed:

amazon-ebs: installing rdoc: /usr/share/ri/2.5.0/system
2019/09/11 12:13:15 packer: 2019/09/11 12:13:15 [ERROR] Remote command exited without exit status or exit signal.

OOM ?

umerebryx · 2019-09-11T10:22:06Z

I will setup a temp repo for you to reproduce this. I have done another test by running same build script from an ec2 instance c4.2xlarge and this time everything completed without any issues. I will double check my local setup to see this is not a memory issue as you mentioned (but shouldn't OOM through a segmentation fault locally?).

anthonymag · 2019-09-23T14:40:12Z

This is happening for me every other build or so, it's pretty inconsistent.
Always seems to happen while installing software-properties-common
Last three failures:

    amazon-ebs: + apt-get install software-properties-common -y
    amazon-ebs: Reading package lists...
    amazon-ebs: Building dependency tree...
    amazon-ebs: Reading state information...
    amazon-ebs: The following additional packages will be installed:
    amazon-ebs:   python3-software-properties
    amazon-ebs: The following packages will be upgraded:
    amazon-ebs:   python3-software-properties software-properties-common
    amazon-ebs: 2 upgraded, 0 newly installed, 0 to remove and 77 not upgraded.
    amazon-ebs: Need to get 29.6 kB of archives.
    amazon-ebs: After this operation, 0 B of additional disk space will be used.
    amazon-ebs: Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 software-properties-common all 0.96.20.9 [9,452 B]
    amazon-ebs: Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3-software-properties all 0.96.20.9 [20.1 kB]
    amazon-ebs: Fetched 29.6 kB in 0s (2,446 kB/s)
    amazon-ebs: (Reading database ... 87051 files and directories currently installed.)
    amazon-ebs: Preparing to unpack .../software-properties-common_0.96.20.9_all.deb ...
    amazon-ebs: Unpacking software-properties-common (0.96.20.9) over (0.96.20.8) ...
    amazon-ebs: Preparing to unpack .../python3-software-properties_0.96.20.9_all.deb ...
    amazon-ebs: Unpacking python3-software-properties (0.96.20.9) over (0.96.20.8) ...
    amazon-ebs: Processing triggers for man-db (2.7.5-1) ...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
Build 'amazon-ebs' errored: Script disconnected unexpectedly. If you expected your script to disconnect, i.e. from a restart, you can try adding `"expect_disconnect": true` to the shell provisioner parameters.

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Script disconnected unexpectedly. If you expected your script to disconnect, i.e. from a restart, you can try adding `"expect_disconnect": true` to the shell provisioner parameters.

==> Builds finished but no artifacts were created.
Build step 'Execute shell' marked build as failure
Finished: FAILURE

    amazon-ebs: + DEBIAN_FRONTEND=noninteractive
    amazon-ebs: + apt-get install software-properties-common -y
    amazon-ebs: Reading package lists...
    amazon-ebs: Building dependency tree...
    amazon-ebs: Reading state information...
    amazon-ebs: The following additional packages will be installed:
    amazon-ebs:   python3-software-properties
    amazon-ebs: The following packages will be upgraded:
    amazon-ebs:   python3-software-properties software-properties-common
    amazon-ebs: 2 upgraded, 0 newly installed, 0 to remove and 77 not upgraded.
    amazon-ebs: Need to get 29.6 kB of archives.
    amazon-ebs: After this operation, 0 B of additional disk space will be used.
    amazon-ebs: Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 software-properties-common all 0.96.20.9 [9,452 B]
    amazon-ebs: Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3-software-properties all 0.96.20.9 [20.1 kB]
    amazon-ebs: Fetched 29.6 kB in 0s (1,911 kB/s)
    amazon-ebs: (Reading database ... 87051 files and directories currently installed.)
    amazon-ebs: Preparing to unpack .../software-properties-common_0.96.20.9_all.deb ...
==> amazon-ebs: Script disconnected unexpectedly. If you expected your script to disconnect, i.e. from a restart, you can try adding `"expect_disconnect": true` to the shell provisioner parameters.
==> amazon-ebs: Step "StepProvision" failed, aborting...
Build 'amazon-ebs' errored: unexpected EOF

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: unexpected EOF

==> Builds finished but no artifacts were created.
Build step 'Execute shell' marked build as failure
Finished: FAILURE

    amazon-ebs: + export DEBIAN_FRONTEND=noninteractive
    amazon-ebs: + DEBIAN_FRONTEND=noninteractive
    amazon-ebs: + apt-get install software-properties-common -y
    amazon-ebs: Reading package lists...
    amazon-ebs: Building dependency tree...
    amazon-ebs: Reading state information...
    amazon-ebs: The following additional packages will be installed:
    amazon-ebs:   python3-software-properties
    amazon-ebs: The following packages will be upgraded:
    amazon-ebs:   python3-software-properties software-properties-common
    amazon-ebs: 2 upgraded, 0 newly installed, 0 to remove and 77 not upgraded.
    amazon-ebs: Need to get 29.6 kB of archives.
    amazon-ebs: After this operation, 0 B of additional disk space will be used.
    amazon-ebs: Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 software-properties-common all 0.96.20.9 [9,452 B]
    amazon-ebs: Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3-software-properties all 0.96.20.9 [20.1 kB]
    amazon-ebs: Fetched 29.6 kB in 0s (2,694 kB/s)
    amazon-ebs: (Reading database ... 87051 files and directories currently installed.)
    amazon-ebs: Preparing to unpack .../software-properties-common_0.96.20.9_all.deb ...
    amazon-ebs: Unpacking software-properties-common (0.96.20.9) over (0.96.20.8) ...
    amazon-ebs: Preparing to unpack .../python3-software-properties_0.96.20.9_all.deb ...
    amazon-ebs: Unpacking python3-software-properties (0.96.20.9) over (0.96.20.8) ...
    amazon-ebs: Processing triggers for man-db (2.7.5-1) ...
    amazon-ebs: Processing triggers for dbus (1.10.6-1ubuntu3.4) ...
    amazon-ebs: Setting up python3-software-properties (0.96.20.9) ...
    amazon-ebs: Setting up software-properties-common (0.96.20.9) ...
==> amazon-ebs: Script disconnected unexpectedly. If you expected your script to disconnect, i.e. from a restart, you can try adding `"expect_disconnect": true` to the shell provisioner parameters.
==> amazon-ebs: Step "StepProvision" failed, aborting...
Build 'amazon-ebs' errored: unexpected EOF

umerebryx · 2019-09-23T18:13:51Z

@anthonymag , for me the issue was fix after I freed up some memory. The worst part is packer doesn't give any indication the termination is because of OOM. You can try running by closing all your programs to see if the issue is RAM in your case. Hope this helps.

anthonymag · 2019-09-23T18:20:29Z

@umerebryx thanks for your response! I'm surprised as I'm giving the built instance 8GB, but I can increase this and try it out, unless you mean memory on the Packer host that is remotely controlling the build?

umerebryx · 2019-09-23T18:24:57Z

You have to free up memory on the host that is running the packer build command and not the remote host.

yaniveliash · 2019-09-23T18:26:37Z

An interesting twist in the story...
I would love to know that all the issue was memory related - please keep us updated if that solves that issue, thanks.

anthonymag · 2019-09-23T18:54:22Z

I'm currently running the build directly from a Jenkins host, and will change it to run from a Jenkins worker/slave to have its own dedicated resources and will let you guys know if that fixes the issue. It might have failed from OOM whenever it was running alongside another build.

anthonymag · 2019-10-07T20:46:54Z

No dice. I gave it its own dedicated host with 8GB and I still get the same failure every other build or so...

proppy · 2022-03-31T02:31:48Z

also got similar issue with long running "shell" provisionner that perform a lot of network i/o (conda) that end up mid-way with:

2022/03/31 10:37:55 packer-plugin-googlecompute_v1.0.11_x5.0_linux_amd64 plugin: 2022/03/31 10:37:55 [ERROR] Remote command exited without exit status or exit signal.

without the actual error being logged.

running packer -debug, inspecting the instance and trying to run the provisioning script manually allowed me to diagnose the error and pin point which command was failing.

tanmay-hc · 2025-03-10T11:28:59Z

@yaniveliash, I have been trying to use all the suggested reproduction steps mentioned in the discussion, however, I cannot reproduce this issue. Maybe it was just a memory issue as indicated by a user.
Please confirm if we can close this ticket.

SwampDragons added bug provisioner/shell communicator/ssh labels Feb 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Script disconnected unexpectedly while long curl #7323

Script disconnected unexpectedly while long curl #7323

yaniveliash commented Feb 18, 2019

yaniveliash commented Feb 20, 2019

umerebryx commented Sep 11, 2019

yaniveliash commented Sep 11, 2019

azr commented Sep 11, 2019 •

edited

Loading

yaniveliash commented Sep 11, 2019

azr commented Sep 11, 2019

umerebryx commented Sep 11, 2019

azr commented Sep 11, 2019 •

edited

Loading

umerebryx commented Sep 11, 2019 •

edited

Loading

anthonymag commented Sep 23, 2019

umerebryx commented Sep 23, 2019

anthonymag commented Sep 23, 2019

umerebryx commented Sep 23, 2019

yaniveliash commented Sep 23, 2019

anthonymag commented Sep 23, 2019

anthonymag commented Oct 7, 2019

proppy commented Mar 31, 2022 •

edited

Loading

tanmay-hc commented Mar 10, 2025

Script disconnected unexpectedly while long curl #7323

Script disconnected unexpectedly while long curl #7323

Comments

yaniveliash commented Feb 18, 2019

yaniveliash commented Feb 20, 2019

umerebryx commented Sep 11, 2019

yaniveliash commented Sep 11, 2019

azr commented Sep 11, 2019 • edited Loading

yaniveliash commented Sep 11, 2019

azr commented Sep 11, 2019

umerebryx commented Sep 11, 2019

azr commented Sep 11, 2019 • edited Loading

umerebryx commented Sep 11, 2019 • edited Loading

anthonymag commented Sep 23, 2019

umerebryx commented Sep 23, 2019

anthonymag commented Sep 23, 2019

umerebryx commented Sep 23, 2019

yaniveliash commented Sep 23, 2019

anthonymag commented Sep 23, 2019

anthonymag commented Oct 7, 2019

proppy commented Mar 31, 2022 • edited Loading

tanmay-hc commented Mar 10, 2025

azr commented Sep 11, 2019 •

edited

Loading

azr commented Sep 11, 2019 •

edited

Loading

umerebryx commented Sep 11, 2019 •

edited

Loading

proppy commented Mar 31, 2022 •

edited

Loading