Discussion:
Katello 3.0.2 (Saison) Released
(too old to reply)
Eric D Helms
2016-07-06 20:51:43 UTC
Permalink
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).

Installation
============

For installation, please see the instructions at:

Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>

Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.

http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Edgars M.
2016-07-07 09:53:23 UTC
Permalink
Hi

I still cannot install new Katello because of this
issue: http://projects.theforeman.org/issues/15507

It fails every time.. Does anyone know workaround?

Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Eric D Helms
2016-07-07 12:07:26 UTC
Permalink
Edgars,

I will test this today and report back to you.


Eric
Post by Edgars M.
Hi
http://projects.theforeman.org/issues/15507
It fails every time.. Does anyone know workaround?
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Eric D Helms
2016-07-07 18:18:50 UTC
Permalink
Edgars,

I tested this scenario today and could not duplicate your results. Is there
anything special about your custom certificates? Wildcard? Attributes
special to them? This is my test scenario:

https://github.com/Katello/forklift/pull/247/files
Post by Eric D Helms
Edgars,
I will test this today and report back to you.
Eric
Post by Edgars M.
Hi
http://projects.theforeman.org/issues/15507
It fails every time.. Does anyone know workaround?
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Edgars M.
2016-07-08 09:12:34 UTC
Permalink
No, there is nothing special about our certificates. No wildcard, no even
SAN. 2048 bits.

Why is Crane needed? Can I disable it? Can I disable everything related to
Puppet as we don't need that functionality?

I just tested it and it failed again, here is my full install command:

sudo foreman-installer --scenario katello \
--certs-server-cert="/etc/pki/tls/certs/katello.tld.crt" \
--certs-server-cert-req="/etc/pki/tls/csr/katello.tld.csr" \
--certs-server-key="/etc/pki/tls/private/katello.tld.key" \
--certs-server-ca-cert="/etc/pki/tls/certs/CompanyInternalCA.crt" \
--foreman-admin-email="***@company.tld" \
--foreman-admin-first-name="Name" \
--foreman-admin-last-name="LastName" \
--foreman-admin-password="SomeCustomPassword" \
--foreman-initial-organization="Company" \
--katello-num-pulp-workers="24" \
--katello-proxy-url="http://corporate.proxy.tld" \
--katello-proxy-port="8080" \
--verbose


Errors:
[ERROR 2016-07-08 10:48:00 verbose] Could not start Service[httpd]:
Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.service
[ INFO 2016-07-08 10:48:00 verbose] Job for httpd.service failed because
the control process exited with error code. See "systemctl status
httpd.service" and "journalctl -xe" for details.
[ERROR 2016-07-08 10:48:00 verbose] /Stage[main]/Apache::Service/Service[
httpd]/ensure: change from stopped to running failed: Could not start
Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.service
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/Foreman
::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/
sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/Foreman
::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed
returned 1 instead of one of [0]
[ERROR 2016-07-08 10:49:15 verbose] /Stage[main]/Foreman_proxy::Register/
Foreman_smartproxy[katello.tld]: Failed to call refresh: Proxy katello.tld
cannot be registered (Could not load data from https://katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when using
apipie cache? (typical production settings)): N/A
[ERROR 2016-07-08 10:49:15 verbose] /Stage[main]/Foreman_proxy::Register/
Foreman_smartproxy[katello.tld]: Proxy katello.tld cannot be registered (
Could not load data from https://katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when using
apipie cache? (typical production settings)): N/A
[ INFO 2016-07-08 10:49:15 verbose] /usr/share/ruby/vendor_ruby/puppet/util/
errors.rb:106:in `fail'
[ INFO 2016-07-08 10:49:19 verbose] Executing hooks in group post
Something went wrong! Check the log for ERROR-level output

sudo cat /var/log/httpd/crane_error_ssl.log
[Fri Jul 08 10:48:00.480289 2016] [ssl:emerg] [pid 13049] AH01903: Failed
to configure CA certificate chain!
[Fri Jul 08 10:57:44.197492 2016] [ssl:emerg] [pid 13508] AH01903: Failed
to configure CA certificate chain!

Edgars
Post by Eric D Helms
Edgars,
I tested this scenario today and could not duplicate your results. Is
there anything special about your custom certificates? Wildcard? Attributes
https://github.com/Katello/forklift/pull/247/files
Post by Eric D Helms
Edgars,
I will test this today and report back to you.
Eric
Post by Edgars M.
Hi
http://projects.theforeman.org/issues/15507
It fails every time.. Does anyone know workaround?
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google
Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send
<javascript:>.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Eric D Helms
2016-07-08 10:58:36 UTC
Permalink
What is the journal -xn output after you try to start httpd manually?
Post by Edgars M.
No, there is nothing special about our certificates. No wildcard, no even
SAN. 2048 bits.
Why is Crane needed? Can I disable it? Can I disable everything related to
Puppet as we don't need that functionality?
sudo foreman-installer --scenario katello \
--certs-server-cert="/etc/pki/tls/certs/katello.tld.crt" \
--certs-server-cert-req="/etc/pki/tls/csr/katello.tld.csr" \
--certs-server-key="/etc/pki/tls/private/katello.tld.key" \
--certs-server-ca-cert="/etc/pki/tls/certs/CompanyInternalCA.crt" \
--foreman-admin-first-name="Name" \
--foreman-admin-last-name="LastName" \
--foreman-admin-password="SomeCustomPassword" \
--foreman-initial-organization="Company" \
--katello-num-pulp-workers="24" \
--katello-proxy-url="http://corporate.proxy.tld" \
--katello-proxy-port="8080" \
--verbose
Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.
service
[ INFO 2016-07-08 10:48:00 verbose] Job for httpd.service failed because
the control process exited with error code. See "systemctl status
httpd.service" and "journalctl -xe" for details.
[ERROR 2016-07-08 10:48:00 verbose] /Stage[main]/Apache::Service/Service[
httpd]/ensure: change from stopped to running failed: Could not start
Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.
service
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/
/usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/
Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake
db:seed returned 1 instead of one of [0]
[ERROR 2016-07-08 10:49:15 verbose] /Stage[main]/Foreman_proxy::Register/
Foreman_smartproxy[katello.tld]: Failed to call refresh: Proxy katello.tld
cannot be registered (Could not load data from https://katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when
using apipie cache? (typical production settings)): N/A
[ERROR 2016-07-08 10:49:15 verbose] /Stage[main]/Foreman_proxy::Register/
Foreman_smartproxy[katello.tld]: Proxy katello.tld cannot be registered (
Could not load data from https://katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when
using apipie cache? (typical production settings)): N/A
[ INFO 2016-07-08 10:49:15 verbose] /usr/share/ruby/vendor_ruby/puppet/
util/errors.rb:106:in `fail'
[ INFO 2016-07-08 10:49:19 verbose] Executing hooks in group post
Something went wrong! Check the log for ERROR-level output
sudo cat /var/log/httpd/crane_error_ssl.log
[Fri Jul 08 10:48:00.480289 2016] [ssl:emerg] [pid 13049] AH01903: Failed
to configure CA certificate chain!
[Fri Jul 08 10:57:44.197492 2016] [ssl:emerg] [pid 13508] AH01903: Failed
to configure CA certificate chain!
Edgars
Post by Eric D Helms
Edgars,
I tested this scenario today and could not duplicate your results. Is
there anything special about your custom certificates? Wildcard? Attributes
https://github.com/Katello/forklift/pull/247/files
Post by Eric D Helms
Edgars,
I will test this today and report back to you.
Eric
Post by Edgars M.
Hi
http://projects.theforeman.org/issues/15507
It fails every time.. Does anyone know workaround?
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google
Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Edgars M.
2016-07-08 11:05:34 UTC
Permalink
Hi

Here it is:

sudo /bin/systemctl start httpd.service
Job for httpd.service failed because the control process exited with error
code. See "systemctl status httpd.service" and "journalctl -xe" for details.

sudo /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor
preset: disabled)
Active: failed (Result: exit-code) since Fri 2016-07-08 10:57:44 CEST; 6s
ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 13510 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1
/FAILURE)
Process: 13508 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=
exited, status=1/FAILURE)
Main PID: 13508 (code=exited, status=1/FAILURE)


Jul 08 10:57:44 katello.tld systemd[1]: Starting The Apache HTTP Server...
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service: main process exited,
code=exited, status=1/FAILURE
Jul 08 10:57:44 katello.tld kill[13510]: kill: cannot find process ""
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service: control process
exited, code=exited status=1
Jul 08 10:57:44 katello.tld systemd[1]: Failed to start The Apache HTTP
Server.
Jul 08 10:57:44 katello.tld systemd[1]: Unit httpd.service entered failed
state.
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service failed.

sudo cat /var/log/httpd/error_log
[Fri Jul 08 10:48:00.407263 2016] [core:notice] [pid 13049] SELinux policy
enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Jul 08 10:48:00.407385 2016] [mime_magic:error] [pid 13049] (2)No such
file or directory: AH01515: mod_mime_magic: can't read magic file
/etc/httpd/NEVER_EVER_USE
[Fri Jul 08 10:48:00.480310 2016] [ssl:emerg] [pid 13049] AH02311: Fatal
error initialising mod_ssl, exiting. See /var/log/httpd/crane_error_ssl.log
for more information
[Fri Jul 08 10:57:44.170768 2016] [core:notice] [pid 13508] SELinux policy
enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Jul 08 10:57:44.170854 2016] [mime_magic:error] [pid 13508] (2)No such
file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd
/NEVER_EVER_USE
[Fri Jul 08 10:57:44.197505 2016] [ssl:emerg] [pid 13508] AH02311: Fatal
error initialising mod_ssl, exiting. See /var/log/httpd/crane_error_ssl.log
for more information

sudo cat /var/log/httpd/crane_error_ssl.log
[Fri Jul 08 10:48:00.480289 2016] [ssl:emerg] [pid 13049] AH01903: Failed
to configure CA certificate chain!
[Fri Jul 08 10:57:44.197492 2016] [ssl:emerg] [pid 13508] AH01903: Failed
to configure CA certificate chain!

sudo cat /etc/httpd/conf.d/03-crane.conf
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************


<VirtualHost *:5000>
ServerName katello.tld


## Vhost docroot
DocumentRoot "/usr/share/crane/"


## Directories, there should at least be a declaration for
/usr/share/crane/


<Directory "/usr/share/crane/">
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Require all granted
</Directory>


## Logging
ErrorLog "/var/log/httpd/crane_error_ssl.log"
ServerSignature Off
CustomLog "/var/log/httpd/crane_access_ssl.log" combined


## SSL directives
SSLEngine on
SSLCertificateFile "/etc/pki/katello/certs/katello-apache.crt"
SSLCertificateKeyFile "/etc/pki/katello/private/katello-apache.key"
SSLCertificateChainFile "/etc/pki/katello/certs/katello-default-ca.crt"
SSLCACertificatePath "/etc/pki/tls/certs"
SSLCACertificateFile "/etc/pki/katello/certs/katello-default-ca.crt"
SSLVerifyClient optional
SSLVerifyDepth 3
SSLOptions +StdEnvVars +ExportCertData +FakeBasicAuth


# SSL Proxy directives
SSLProxyEngine On
WSGIScriptAlias / "/usr/share/crane/crane.wsgi"
</VirtualHost>
Post by Eric D Helms
What is the journal -xn output after you try to start httpd manually?
Post by Edgars M.
No, there is nothing special about our certificates. No wildcard, no even
SAN. 2048 bits.
Why is Crane needed? Can I disable it? Can I disable everything related
to Puppet as we don't need that functionality?
sudo foreman-installer --scenario katello \
--certs-server-cert="/etc/pki/tls/certs/katello.tld.crt" \
--certs-server-cert-req="/etc/pki/tls/csr/katello.tld.csr" \
--certs-server-key="/etc/pki/tls/private/katello.tld.key" \
--certs-server-ca-cert="/etc/pki/tls/certs/CompanyInternalCA.crt" \
--foreman-admin-first-name="Name" \
--foreman-admin-last-name="LastName" \
--foreman-admin-password="SomeCustomPassword" \
--foreman-initial-organization="Company" \
--katello-num-pulp-workers="24" \
--katello-proxy-url="http://corporate.proxy.tld" \
--katello-proxy-port="8080" \
--verbose
Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.
service
[ INFO 2016-07-08 10:48:00 verbose] Job for httpd.service failed because
the control process exited with error code. See "systemctl status
httpd.service" and "journalctl -xe" for details.
[ERROR 2016-07-08 10:48:00 verbose] /Stage[main]/Apache::Service/Service
[httpd]/ensure: change from stopped to running failed: Could not start
Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.
service
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/
Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh
: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/
Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake
db:seed returned 1 instead of one of [0]
[ERROR 2016-07-08 10:49:15 verbose] /Stage[main]/Foreman_proxy::Register
/Foreman_smartproxy[katello.tld]: Failed to call refresh: Proxy katello.tld
cannot be registered (Could not load data from https://katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when
using apipie cache? (typical production settings)): N/A
[ERROR 2016-07-08 10:49:15 verbose] /Stage[main]/Foreman_proxy::Register
/Foreman_smartproxy[katello.tld]: Proxy katello.tld cannot be registered
(Could not load data from https://katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when
using apipie cache? (typical production settings)): N/A
[ INFO 2016-07-08 10:49:15 verbose] /usr/share/ruby/vendor_ruby/puppet/
util/errors.rb:106:in `fail'
[ INFO 2016-07-08 10:49:19 verbose] Executing hooks in group post
Something went wrong! Check the log for ERROR-level output
sudo cat /var/log/httpd/crane_error_ssl.log
[Fri Jul 08 10:48:00.480289 2016] [ssl:emerg] [pid 13049] AH01903: Failed
to configure CA certificate chain!
[Fri Jul 08 10:57:44.197492 2016] [ssl:emerg] [pid 13508] AH01903: Failed
to configure CA certificate chain!
Edgars
Post by Eric D Helms
Edgars,
I tested this scenario today and could not duplicate your results. Is
there anything special about your custom certificates? Wildcard? Attributes
https://github.com/Katello/forklift/pull/247/files
Post by Eric D Helms
Edgars,
I will test this today and report back to you.
Eric
Post by Edgars M.
Hi
http://projects.theforeman.org/issues/15507
It fails every time.. Does anyone know workaround?
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google
Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Eric D Helms
2016-07-08 11:09:28 UTC
Permalink
Can you run:

systemctl start httpd
journalctl -xn
Post by Edgars M.
Hi
sudo /bin/systemctl start httpd.service
Job for httpd.service failed because the control process exited with
error code. See "systemctl status httpd.service" and "journalctl -xe" for
details.
sudo /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled;
vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2016-07-08 10:57:44 CEST;
6s ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 13510 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status
=1/FAILURE)
Process: 13508 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=
exited, status=1/FAILURE)
Main PID: 13508 (code=exited, status=1/FAILURE)
Jul 08 10:57:44 katello.tld systemd[1]: Starting The Apache HTTP Server...
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service: main process exited
, code=exited, status=1/FAILURE
Jul 08 10:57:44 katello.tld kill[13510]: kill: cannot find process ""
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service: control process
exited, code=exited status=1
Jul 08 10:57:44 katello.tld systemd[1]: Failed to start The Apache HTTP
Server.
Jul 08 10:57:44 katello.tld systemd[1]: Unit httpd.service entered failed
state.
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service failed.
sudo cat /var/log/httpd/error_log
[Fri Jul 08 10:48:00.407263 2016] [core:notice] [pid 13049] SELinux
policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Jul 08 10:48:00.407385 2016] [mime_magic:error] [pid 13049] (2)No
such file or directory: AH01515: mod_mime_magic: can't read magic file
/etc/httpd/NEVER_EVER_USE
[Fri Jul 08 10:48:00.480310 2016] [ssl:emerg] [pid 13049] AH02311: Fatal
error initialising mod_ssl, exiting. See /var/log/httpd/crane_error_ssl.log
for more information
[Fri Jul 08 10:57:44.170768 2016] [core:notice] [pid 13508] SELinux policy
enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Jul 08 10:57:44.170854 2016] [mime_magic:error] [pid 13508] (2)No
such file or directory: AH01515: mod_mime_magic: can't read magic file /
etc/httpd/NEVER_EVER_USE
[Fri Jul 08 10:57:44.197505 2016] [ssl:emerg] [pid 13508] AH02311: Fatal
error initialising mod_ssl, exiting. See /var/log/httpd/crane_error_ssl.log
for more information
sudo cat /var/log/httpd/crane_error_ssl.log
[Fri Jul 08 10:48:00.480289 2016] [ssl:emerg] [pid 13049] AH01903: Failed
to configure CA certificate chain!
[Fri Jul 08 10:57:44.197492 2016] [ssl:emerg] [pid 13508] AH01903: Failed
to configure CA certificate chain!
sudo cat /etc/httpd/conf.d/03-crane.conf
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************
<VirtualHost *:5000>
ServerName katello.tld
## Vhost docroot
DocumentRoot "/usr/share/crane/"
## Directories, there should at least be a declaration for
/usr/share/crane/
<Directory "/usr/share/crane/">
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Require all granted
</Directory>
## Logging
ErrorLog "/var/log/httpd/crane_error_ssl.log"
ServerSignature Off
CustomLog "/var/log/httpd/crane_access_ssl.log" combined
## SSL directives
SSLEngine on
SSLCertificateFile "/etc/pki/katello/certs/katello-apache.crt"
SSLCertificateKeyFile "/etc/pki/katello/private/katello-apache.key"
SSLCertificateChainFile "/etc/pki/katello/certs/katello-default-ca.crt"
SSLCACertificatePath "/etc/pki/tls/certs"
SSLCACertificateFile "/etc/pki/katello/certs/katello-default-ca.crt"
SSLVerifyClient optional
SSLVerifyDepth 3
SSLOptions +StdEnvVars +ExportCertData +FakeBasicAuth
# SSL Proxy directives
SSLProxyEngine On
WSGIScriptAlias / "/usr/share/crane/crane.wsgi"
</VirtualHost>
Post by Eric D Helms
What is the journal -xn output after you try to start httpd manually?
Post by Edgars M.
No, there is nothing special about our certificates. No wildcard, no
even SAN. 2048 bits.
Why is Crane needed? Can I disable it? Can I disable everything related
to Puppet as we don't need that functionality?
sudo foreman-installer --scenario katello \
--certs-server-cert="/etc/pki/tls/certs/katello.tld.crt" \
--certs-server-cert-req="/etc/pki/tls/csr/katello.tld.csr" \
--certs-server-key="/etc/pki/tls/private/katello.tld.key" \
--certs-server-ca-cert="/etc/pki/tls/certs/CompanyInternalCA.crt" \
--foreman-admin-first-name="Name" \
--foreman-admin-last-name="LastName" \
--foreman-admin-password="SomeCustomPassword" \
--foreman-initial-organization="Company" \
--katello-num-pulp-workers="24" \
--katello-proxy-url="http://corporate.proxy.tld" \
--katello-proxy-port="8080" \
--verbose
Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.
service
[ INFO 2016-07-08 10:48:00 verbose] Job for httpd.service failed
because the control process exited with error code. See "systemctl
status httpd.service" and "journalctl -xe" for details.
[ERROR 2016-07-08 10:48:00 verbose] /Stage[main]/Apache::Service/
Service[httpd]/ensure: change from stopped to running failed: Could not
start Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.
service
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/
Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call
refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/
Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake
db:seed returned 1 instead of one of [0]
Register/Foreman_smartproxy[katello.tld]: Failed to call refresh: Proxy
//katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when
using apipie cache? (typical production settings)): N/A
Register/Foreman_smartproxy[katello.tld]: Proxy katello.tld cannot be
registered (Could not load data from https://katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when
using apipie cache? (typical production settings)): N/A
[ INFO 2016-07-08 10:49:15 verbose] /usr/share/ruby/vendor_ruby/puppet/
util/errors.rb:106:in `fail'
[ INFO 2016-07-08 10:49:19 verbose] Executing hooks in group post
Something went wrong! Check the log for ERROR-level output
sudo cat /var/log/httpd/crane_error_ssl.log
Failed to configure CA certificate chain!
Failed to configure CA certificate chain!
Edgars
Post by Eric D Helms
Edgars,
I tested this scenario today and could not duplicate your results. Is
there anything special about your custom certificates? Wildcard? Attributes
https://github.com/Katello/forklift/pull/247/files
Post by Eric D Helms
Edgars,
I will test this today and report back to you.
Eric
Post by Edgars M.
Hi
http://projects.theforeman.org/issues/15507
It fails every time.. Does anyone know workaround?
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major
upgrade issues found by some awesome users. Please see the changelog for
more information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google
Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google
Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Edgars M.
2016-07-08 11:26:55 UTC
Permalink
sudo systemctl start httpd
Job for httpd.service failed because the control process exited with error
code. See "systemctl status httpd.service" and "journalctl -xe" for details.

sudo journalctl -xn
-- Logs begin at Thu 2016-07-07 11:09:00 CEST, end at Fri 2016-07-08 13:19:
56 CEST. --
Jul 08 13:19:44 katello.tld polkitd[1437]: Registered Authentication Agent
for unix-process:13638:8287367 (system bus name :1.285 [/usr/bin/pkttyagent
--notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/Authenti
Jul 08 13:19:44 katello.tld systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has begun starting up.
Jul 08 13:19:44 katello.tld systemd[1]: httpd.service: main process exited,
code=exited, status=1/FAILURE
Jul 08 13:19:44 katello.tld kill[13645]: kill: cannot find process ""
Jul 08 13:19:44 katello.tld systemd[1]: httpd.service: control process
exited, code=exited status=1
Jul 08 13:19:44 katello.tld systemd[1]: Failed to start The Apache HTTP
Server.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has failed.
--
-- The result is failed.
Jul 08 13:19:44 katello.tld systemd[1]: Unit httpd.service entered failed
state.
Jul 08 13:19:44 katello.tld systemd[1]: httpd.service failed.
Jul 08 13:19:44 katello.tld polkitd[1437]: Unregistered Authentication Agent
for unix-process:13638:8287367 (system bus name :1.285, object path /org/
freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnecte
Jul 08 13:19:56 katello.tld sudo[13649]: edgars : TTY=pts/0 ; PWD=/home/
edgars ; USER=root ; COMMAND=/bin/journalctl -xn
Post by Eric D Helms
systemctl start httpd
journalctl -xn
Post by Edgars M.
Hi
sudo /bin/systemctl start httpd.service
Job for httpd.service failed because the control process exited with
error code. See "systemctl status httpd.service" and "journalctl -xe" for
details.
sudo /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled;
vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2016-07-08 10:57:44 CEST;
6s ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 13510 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited,
status=1/FAILURE)
Process: 13508 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=
exited, status=1/FAILURE)
Main PID: 13508 (code=exited, status=1/FAILURE)
Jul 08 10:57:44 katello.tld systemd[1]: Starting The Apache HTTP Server
...
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service: main process
exited, code=exited, status=1/FAILURE
Jul 08 10:57:44 katello.tld kill[13510]: kill: cannot find process ""
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service: control process
exited, code=exited status=1
Jul 08 10:57:44 katello.tld systemd[1]: Failed to start The Apache HTTP
Server.
Jul 08 10:57:44 katello.tld systemd[1]: Unit httpd.service entered
failed state.
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service failed.
sudo cat /var/log/httpd/error_log
[Fri Jul 08 10:48:00.407263 2016] [core:notice] [pid 13049] SELinux
policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Jul 08 10:48:00.407385 2016] [mime_magic:error] [pid 13049] (2)No
such file or directory: AH01515: mod_mime_magic: can't read magic file
/etc/httpd/NEVER_EVER_USE
[Fri Jul 08 10:48:00.480310 2016] [ssl:emerg] [pid 13049] AH02311: Fatal
error initialising mod_ssl, exiting. See /var/log/httpd/crane_error_ssl.log
for more information
[Fri Jul 08 10:57:44.170768 2016] [core:notice] [pid 13508] SELinux
policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Jul 08 10:57:44.170854 2016] [mime_magic:error] [pid 13508] (2)No
such file or directory: AH01515: mod_mime_magic: can't read magic file /
etc/httpd/NEVER_EVER_USE
[Fri Jul 08 10:57:44.197505 2016] [ssl:emerg] [pid 13508] AH02311: Fatal
error initialising mod_ssl, exiting. See /var/log/httpd/crane_error_ssl.log
for more information
sudo cat /var/log/httpd/crane_error_ssl.log
[Fri Jul 08 10:48:00.480289 2016] [ssl:emerg] [pid 13049] AH01903: Failed
to configure CA certificate chain!
[Fri Jul 08 10:57:44.197492 2016] [ssl:emerg] [pid 13508] AH01903: Failed
to configure CA certificate chain!
sudo cat /etc/httpd/conf.d/03-crane.conf
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************
<VirtualHost *:5000>
ServerName katello.tld
## Vhost docroot
DocumentRoot "/usr/share/crane/"
## Directories, there should at least be a declaration for
/usr/share/crane/
<Directory "/usr/share/crane/">
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Require all granted
</Directory>
## Logging
ErrorLog "/var/log/httpd/crane_error_ssl.log"
ServerSignature Off
CustomLog "/var/log/httpd/crane_access_ssl.log" combined
## SSL directives
SSLEngine on
SSLCertificateFile "/etc/pki/katello/certs/katello-apache.crt"
SSLCertificateKeyFile "/etc/pki/katello/private/katello-apache.key"
SSLCertificateChainFile "/etc/pki/katello/certs/katello-default-ca.crt"
SSLCACertificatePath "/etc/pki/tls/certs"
SSLCACertificateFile "/etc/pki/katello/certs/katello-default-ca.crt"
SSLVerifyClient optional
SSLVerifyDepth 3
SSLOptions +StdEnvVars +ExportCertData +FakeBasicAuth
# SSL Proxy directives
SSLProxyEngine On
WSGIScriptAlias / "/usr/share/crane/crane.wsgi"
</VirtualHost>
Post by Eric D Helms
What is the journal -xn output after you try to start httpd manually?
Post by Edgars M.
No, there is nothing special about our certificates. No wildcard, no
even SAN. 2048 bits.
Why is Crane needed? Can I disable it? Can I disable everything related
to Puppet as we don't need that functionality?
sudo foreman-installer --scenario katello \
--certs-server-cert="/etc/pki/tls/certs/katello.tld.crt" \
--certs-server-cert-req="/etc/pki/tls/csr/katello.tld.csr" \
--certs-server-key="/etc/pki/tls/private/katello.tld.key" \
--certs-server-ca-cert="/etc/pki/tls/certs/CompanyInternalCA.crt" \
--foreman-admin-first-name="Name" \
--foreman-admin-last-name="LastName" \
--foreman-admin-password="SomeCustomPassword" \
--foreman-initial-organization="Company" \
--katello-num-pulp-workers="24" \
--katello-proxy-url="http://corporate.proxy.tld" \
--katello-proxy-port="8080" \
--verbose
Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.
service
[ INFO 2016-07-08 10:48:00 verbose] Job for httpd.service failed
because the control process exited with error code. See "systemctl
status httpd.service" and "journalctl -xe" for details.
[ERROR 2016-07-08 10:48:00 verbose] /Stage[main]/Apache::Service/
Service[httpd]/ensure: change from stopped to running failed: Could not
start Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.
service
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/
Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call
refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0
]
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/
Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake
db:seed returned 1 instead of one of [0]
Register/Foreman_smartproxy[katello.tld]: Failed to call refresh: Proxy
//katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when
using apipie cache? (typical production settings)): N/A
Register/Foreman_smartproxy[katello.tld]: Proxy katello.tld cannot be
registered (Could not load data from https://katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when
using apipie cache? (typical production settings)): N/A
[ INFO 2016-07-08 10:49:15 verbose] /usr/share/ruby/vendor_ruby/puppet/
util/errors.rb:106:in `fail'
[ INFO 2016-07-08 10:49:19 verbose] Executing hooks in group post
Something went wrong! Check the log for ERROR-level output
sudo cat /var/log/httpd/crane_error_ssl.log
Failed to configure CA certificate chain!
Failed to configure CA certificate chain!
Edgars
Post by Eric D Helms
Edgars,
I tested this scenario today and could not duplicate your results. Is
there anything special about your custom certificates? Wildcard? Attributes
https://github.com/Katello/forklift/pull/247/files
Post by Eric D Helms
Edgars,
I will test this today and report back to you.
Eric
Post by Edgars M.
Hi
http://projects.theforeman.org/issues/15507
It fails every time.. Does anyone know workaround?
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major
upgrade issues found by some awesome users. Please see the changelog for
more information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google
Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google
Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Edgars M.
2016-07-08 12:25:30 UTC
Permalink
And just to make sure my certs are ok, I used katello-certs-check tool and
got:

Validating the certificate subject=XXXXXX
Check private key matches the certificate: [OK]
Check ca bundle verifies the cert file: [OK]


Validation succeeded.
Post by Edgars M.
sudo systemctl start httpd
Job for httpd.service failed because the control process exited with
error code. See "systemctl status httpd.service" and "journalctl -xe" for
details.
sudo journalctl -xn
-- Logs begin at Thu 2016-07-07 11:09:00 CEST, end at Fri 2016-07-08 13:19
:56 CEST. --
Jul 08 13:19:44 katello.tld polkitd[1437]: Registered Authentication Agent
for unix-process:13638:8287367 (system bus name :1.285 [/usr/bin/pkttyagent
--notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/
Authenti
Jul 08 13:19:44 katello.tld systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has begun starting up.
Jul 08 13:19:44 katello.tld systemd[1]: httpd.service: main process exited
, code=exited, status=1/FAILURE
Jul 08 13:19:44 katello.tld kill[13645]: kill: cannot find process ""
Jul 08 13:19:44 katello.tld systemd[1]: httpd.service: control process
exited, code=exited status=1
Jul 08 13:19:44 katello.tld systemd[1]: Failed to start The Apache HTTP
Server.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has failed.
--
-- The result is failed.
Jul 08 13:19:44 katello.tld systemd[1]: Unit httpd.service entered failed
state.
Jul 08 13:19:44 katello.tld systemd[1]: httpd.service failed.
Jul 08 13:19:44 katello.tld polkitd[1437]: Unregistered Authentication
Agent for unix-process:13638:8287367 (system bus name :1.285, object path
/org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (
disconnecte
Jul 08 13:19:56 katello.tld sudo[13649]: edgars : TTY=pts/0 ; PWD=
/home/edgars ; USER=root ; COMMAND=/bin/journalctl -xn
Post by Eric D Helms
systemctl start httpd
journalctl -xn
Post by Edgars M.
Hi
sudo /bin/systemctl start httpd.service
Job for httpd.service failed because the control process exited with
error code. See "systemctl status httpd.service" and "journalctl -xe"
for details.
sudo /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled;
vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2016-07-08 10:57:44 CEST
; 6s ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 13510 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited,
status=1/FAILURE)
Process: 13508 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=
exited, status=1/FAILURE)
Main PID: 13508 (code=exited, status=1/FAILURE)
Jul 08 10:57:44 katello.tld systemd[1]: Starting The Apache HTTP Server
...
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service: main process
exited, code=exited, status=1/FAILURE
Jul 08 10:57:44 katello.tld kill[13510]: kill: cannot find process ""
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service: control process
exited, code=exited status=1
Jul 08 10:57:44 katello.tld systemd[1]: Failed to start The Apache HTTP
Server.
Jul 08 10:57:44 katello.tld systemd[1]: Unit httpd.service entered
failed state.
Jul 08 10:57:44 katello.tld systemd[1]: httpd.service failed.
sudo cat /var/log/httpd/error_log
[Fri Jul 08 10:48:00.407263 2016] [core:notice] [pid 13049] SELinux
policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Jul 08 10:48:00.407385 2016] [mime_magic:error] [pid 13049] (2)No
such file or directory: AH01515: mod_mime_magic: can't read magic file
/etc/httpd/NEVER_EVER_USE
[Fri Jul 08 10:48:00.480310 2016] [ssl:emerg] [pid 13049] AH02311: Fatal
error initialising mod_ssl, exiting. See /var/log/httpd/crane_error_ssl.log
for more information
[Fri Jul 08 10:57:44.170768 2016] [core:notice] [pid 13508] SELinux
policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Jul 08 10:57:44.170854 2016] [mime_magic:error] [pid 13508] (2)No
such file or directory: AH01515: mod_mime_magic: can't read magic file /
etc/httpd/NEVER_EVER_USE
[Fri Jul 08 10:57:44.197505 2016] [ssl:emerg] [pid 13508] AH02311: Fatal
error initialising mod_ssl, exiting. See /var/log/httpd/crane_error_ssl.log
for more information
sudo cat /var/log/httpd/crane_error_ssl.log
Failed to configure CA certificate chain!
Failed to configure CA certificate chain!
sudo cat /etc/httpd/conf.d/03-crane.conf
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************
<VirtualHost *:5000>
ServerName katello.tld
## Vhost docroot
DocumentRoot "/usr/share/crane/"
## Directories, there should at least be a declaration for
/usr/share/crane/
<Directory "/usr/share/crane/">
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Require all granted
</Directory>
## Logging
ErrorLog "/var/log/httpd/crane_error_ssl.log"
ServerSignature Off
CustomLog "/var/log/httpd/crane_access_ssl.log" combined
## SSL directives
SSLEngine on
SSLCertificateFile "/etc/pki/katello/certs/katello-apache.crt"
SSLCertificateKeyFile "/etc/pki/katello/private/katello-apache.key"
SSLCertificateChainFile "/etc/pki/katello/certs/katello-default-ca.crt
"
SSLCACertificatePath "/etc/pki/tls/certs"
SSLCACertificateFile "/etc/pki/katello/certs/katello-default-ca.crt
"
SSLVerifyClient optional
SSLVerifyDepth 3
SSLOptions +StdEnvVars +ExportCertData +FakeBasicAuth
# SSL Proxy directives
SSLProxyEngine On
WSGIScriptAlias / "/usr/share/crane/crane.wsgi"
</VirtualHost>
Post by Eric D Helms
What is the journal -xn output after you try to start httpd manually?
Post by Edgars M.
No, there is nothing special about our certificates. No wildcard, no
even SAN. 2048 bits.
Why is Crane needed? Can I disable it? Can I disable everything
related to Puppet as we don't need that functionality?
sudo foreman-installer --scenario katello \
--certs-server-cert="/etc/pki/tls/certs/katello.tld.crt" \
--certs-server-cert-req="/etc/pki/tls/csr/katello.tld.csr" \
--certs-server-key="/etc/pki/tls/private/katello.tld.key" \
--certs-server-ca-cert="/etc/pki/tls/certs/CompanyInternalCA.crt" \
--foreman-admin-first-name="Name" \
--foreman-admin-last-name="LastName" \
--foreman-admin-password="SomeCustomPassword" \
--foreman-initial-organization="Company" \
--katello-num-pulp-workers="24" \
--katello-proxy-url="http://corporate.proxy.tld" \
--katello-proxy-port="8080" \
--verbose
Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.
service
[ INFO 2016-07-08 10:48:00 verbose] Job for httpd.service failed
because the control process exited with error code. See "systemctl
status httpd.service" and "journalctl -xe" for details.
[ERROR 2016-07-08 10:48:00 verbose] /Stage[main]/Apache::Service/
Service[httpd]/ensure: change from stopped to running failed: Could
not start Service[httpd]: Execution of '/usr/share/katello-installer-base/modules/service_wait/bin/service-wait
start httpd' returned 1: Redirecting to /bin/systemctl start httpd.
service
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/
Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call
refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [
0]
[ERROR 2016-07-08 10:48:18 verbose] /Stage[main]/Foreman::Database/
Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake
db:seed returned 1 instead of one of [0]
Proxy katello.tld cannot be registered (Could not load data from https
://katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when
using apipie cache? (typical production settings)): N/A
Register/Foreman_smartproxy[katello.tld]: Proxy katello.tld cannot be
registered (Could not load data from https://katello.tld
[ INFO 2016-07-08 10:49:15 verbose] - is your server down?
[ INFO 2016-07-08 10:49:15 verbose] - was rake apipie:cache run when
using apipie cache? (typical production settings)): N/A
[ INFO 2016-07-08 10:49:15 verbose] /usr/share/ruby/vendor_ruby/puppet
/util/errors.rb:106:in `fail'
[ INFO 2016-07-08 10:49:19 verbose] Executing hooks in group post
Something went wrong! Check the log for ERROR-level output
sudo cat /var/log/httpd/crane_error_ssl.log
Failed to configure CA certificate chain!
Failed to configure CA certificate chain!
Edgars
Post by Eric D Helms
Edgars,
I tested this scenario today and could not duplicate your results. Is
there anything special about your custom certificates? Wildcard? Attributes
https://github.com/Katello/forklift/pull/247/files
Post by Eric D Helms
Edgars,
I will test this today and report back to you.
Eric
Post by Edgars M.
Hi
http://projects.theforeman.org/issues/15507
It fails every time.. Does anyone know workaround?
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major
upgrade issues found by some awesome users. Please see the changelog for
more information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google
Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google
Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Edgars M.
2016-07-26 11:17:55 UTC
Permalink
Hi Eric

Did you have a chance to look in to this issue?

Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Eric D Helms
2016-07-26 11:40:37 UTC
Permalink
I have tried to reproduce this myself to no avail. Anything special about
your custom certs? Can you check if the CA cert is your custom one being
used by crane? Also the server cert? The filenames won't reflect that so
check the contents.
Post by Edgars M.
Hi Eric
Did you have a chance to look in to this issue?
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Edgars M.
2016-07-26 12:35:43 UTC
Permalink
I will check certs again, though they are standard certs used across our
infra

Crane config is configured with default CA certs, not my custom:


## SSL directives
SSLEngine on
SSLCertificateFile "/etc/pki/katello/certs/katello-apache.crt"
SSLCertificateKeyFile "/etc/pki/katello/private/katello-apache.key"
SSLCertificateChainFile "/etc/pki/katello/certs/katello-default-ca.crt"
SSLCACertificatePath "/etc/pki/tls/certs"
SSLCACertificateFile "/etc/pki/katello/certs/katello-default-ca.crt"
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Edgars M.
2016-07-26 13:49:41 UTC
Permalink
Hi Eric

I re-generated all certs and re-downloaded CA cert again and now it works!
I have no idea what was wrong with them. Sorry about all the noise :)

Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Eric D Helms
2016-07-26 14:07:00 UTC
Permalink
No worries! Certs don't always make it easy -- glad to hear its working!

Eric
Post by Edgars M.
Hi Eric
I re-generated all certs and re-downloaded CA cert again and now it works!
I have no idea what was wrong with them. Sorry about all the noise :)
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Abir
2016-07-26 17:48:58 UTC
Permalink
Sounds like an SELinux issue. Ran into something similar when I copied /
generated certificates in my home directory then moved them to /etc/httpd.
If you have the files backed up, you can try running 'ls -alZ' in the
directory to check the file context.
Running 'restorecon <filename>' on your certificate files would have
resolved it if SELinux was the issue.
Post by Eric D Helms
No worries! Certs don't always make it easy -- glad to hear its working!
Eric
Post by Edgars M.
Hi Eric
I re-generated all certs and re-downloaded CA cert again and now it
works! I have no idea what was wrong with them. Sorry about all the noise :)
Edgars
Post by Eric D Helms
Katello 3.0.2 has been released to supply bug fixes and major upgrade
issues found by some awesome users. Please see the changelog for more
information (
https://github.com/Katello/katello/blob/KATELLO-3.0/CHANGELOG.md).
Installation
============
Server: http://www.katello.org/docs/3.0/installation/index.html
<http://www.katello.org/docs/2.4/installation/index.html>
Capsule: http://www.katello.org/docs/3.0/installation/capsule.html
<http://www.katello.org/docs/2.4/installation/capsule.html>
Bug reporting
=============
If you come across a bug in your testing, please file it and note the
version of Katello that you're using in the report and set the release
to 3.0.2.
http://projects.theforeman.org/projects/katello/issues/new
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Rick Langston
2016-08-01 11:54:40 UTC
Permalink
Hello

I seem to be having some odd behavior with this version. With a fresh
install on centos 7 I have setup a product which completes normally but
when I discover a repo and save them i get these meta data task that seem
to just wait forever. Any ideas what can be the culprit


Action:

Actions::Pulp::Repository::DistributorPublish

State: waiting for Pulp to start the task
Input:

{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}

Output:

{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64", "pulp:action:publish"],
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Chris Duryee
2016-08-01 12:12:22 UTC
Permalink
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With a fresh
install on centos 7 I have setup a product which completes normally but
when I discover a repo and save them i get these meta data task that seem
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64", "pulp:action:publish"],
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Rick Langston
2016-08-01 12:50:07 UTC
Permalink
I do see this issue in messages but not sure if its related

Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416) beat
raised exception <class 'qpid.messaging.exceptions.Timeout'>:
Timeout('Connection attach timed out',)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
Traceback (most recent call last):
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416) File
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat.start()
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With a fresh
install on centos 7 I have setup a product which completes normally but
when I discover a repo and save them i get these meta data task that
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Chris Duryee
2016-08-01 13:22:42 UTC
Permalink
Post by Rick Langston
I do see this issue in messages but not sure if its related
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416) beat
Timeout('Connection attach timed out',)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416) File
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat.start()
That is the likely culprit:)

Next time your task hangs, check in the "/about" page on your Katello
instance and ensure everything under "Backend System Status" says "OK"
with no further message.

If there are pulp errors, a possible quick fix is to ensure qpidd is
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up after that.

Also, please check dmesg for out-of-memory errors. There are some other
possible things we can check, but I would be curious first about the
backend system status output.
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With a fresh
install on centos 7 I have setup a product which completes normally but
when I discover a repo and save them i get these meta data task that
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Rick Langston
2016-08-01 13:39:07 UTC
Permalink
The backend service all say ok but when a run a katelli-service status I
can see that celery.bet fails to status. if I restart the server it and
immediately check the status it says running but checking the status again
shows it timed out.


No memory errros noted


[***@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)

Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]: celery.beat:CRITICAL:
(5887-79264) raise Timeout("Connection attach timed out")
Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]: celery.beat:CRITICAL:
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat v3.1.11
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ - ... __
- _
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: Configuration ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
[stderr]@%INFO
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . maxinterval ->
now (0s)
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416) beat
Timeout('Connection attach timed out',)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your Katello
instance and ensure everything under "Backend System Status" says "OK"
with no further message.
If there are pulp errors, a possible quick fix is to ensure qpidd is
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up after that.
Also, please check dmesg for out-of-memory errors. There are some other
possible things we can check, but I would be curious first about the
backend system status output.
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With a fresh
install on centos 7 I have setup a product which completes normally
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data task that
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Chris Duryee
2016-08-01 13:43:24 UTC
Permalink
Post by Rick Langston
The backend service all say ok but when a run a katelli-service status I
can see that celery.bet fails to status. if I restart the server it and
immediately check the status it says running but checking the status again
shows it timed out.
Is this the sequence of events?

* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat v3.1.11
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ - ... __
- _
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: Configuration ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . maxinterval ->
now (0s)
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416) beat
Timeout('Connection attach timed out',)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your Katello
instance and ensure everything under "Backend System Status" says "OK"
with no further message.
If there are pulp errors, a possible quick fix is to ensure qpidd is
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up after that.
Also, please check dmesg for out-of-memory errors. There are some other
possible things we can check, but I would be curious first about the
backend system status output.
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With a fresh
install on centos 7 I have setup a product which completes normally
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data task that
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Rick Langston
2016-08-01 13:49:06 UTC
Permalink
Thank you for help on this

yes here is the actual capture

command line output
[***@dscaprv01 tmp]# systemctl restart pulp_celerybeat.service
[***@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s ago
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler

Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started Pulp's
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting Pulp's
Celerybeat...
Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]: celery.beat:INFO:
beat: Starting...
Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
Aug 01 08:34:25 dscaprv01.corp.acxiom.net pulp[5887]:
pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
Aug 01 08:34:26 dscaprv01.corp.acxiom.net pulp[5887]:
pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
[***@dscaprv01 tmp]# systemctl status pulp_celerybeat.service
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)

Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]: celery.beat:CRITICAL:
(5887-79264) raise Timeout("Connection attach timed out")
Aug 01 08:34:30 dscaprv01.corp.acxiom.net pulp[5887]: celery.beat:CRITICAL:
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat v3.1.11
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ - ... __
- _
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: Configuration ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
[stderr]@%INFO
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . maxinterval ->
now (0s)

/var/log/messages output

==> /var/log/messages <==
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) beat
raised exception <class 'qpid.messaging.exceptions.Timeout'>:
Timeout('Connection attach timed out',)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Traceback (most recent call last):
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat.start()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval = self.scheduler.tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line
265, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) ret
= self.call_tick(self, celerybeat_name)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line
230, in call_tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) ret
= super(Scheduler, self).tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
next_time_to_run = self.maybe_due(entry, self.publisher)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325, in
__get__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
value = obj.__dict__[self.__name__] = self.__get(obj)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in publisher
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.Publisher(self._ensure_connected())
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in
ensure_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval_start, interval_step, interval_max, callback)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in
retry_over_time
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return fun(*args, **kwargs)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in
connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._connection = self._establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in
_establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) conn
= self.transport.establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in
establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) conn
= self.Connection(**opts)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in
__init__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._qpid_conn = establish(**self.connection_options)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112,
in establish
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn.open(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"<string>", line 6, in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323,
in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self.attach(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"<string>", line 6, in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 343,
in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
raise Timeout("Connection attach timed out")
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) Timeout:
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater) is starting.
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader -> celery.loaders.app.AppLoader
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . logfile -> [stderr]@%INFO
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service status I
can see that celery.bet fails to status. if I restart the server it and
immediately check the status it says running but checking the status
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ - ...
__
Post by Rick Langston
- _
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: Configuration
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . maxinterval
->
Post by Rick Langston
now (0s)
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your Katello
instance and ensure everything under "Backend System Status" says "OK"
with no further message.
If there are pulp errors, a possible quick fix is to ensure qpidd is
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up after
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are some other
possible things we can check, but I would be curious first about the
backend system status output.
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With a
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes normally
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data task that
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Chris Duryee
2016-08-01 14:52:19 UTC
Permalink
Post by Rick Langston
Thank you for help on this
yes here is the actual capture
Thanks for the capture of the logs.

What is the output of systemctl status qpidd.service? I am curious if
only celerybeat is unable to connect, or if there is some other issue.

If you want to restart qpidd, the best way is to just do a
katello-service stop. A number of other services depend on qpid, so it's
easiest to just down everything, ensure qpidd is stopped, then bring it
all back up via the script.
Post by Rick Langston
command line output
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s ago
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started Pulp's
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting Pulp's
Celerybeat...
beat: Starting...
pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat v3.1.11
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ - ... __
- _
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: Configuration ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . maxinterval ->
now (0s)
/var/log/messages output
==> /var/log/messages <==
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) beat
Timeout('Connection attach timed out',)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat.start()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval = self.scheduler.tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line
265, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) ret
= self.call_tick(self, celerybeat_name)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line
230, in call_tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) ret
= super(Scheduler, self).tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
next_time_to_run = self.maybe_due(entry, self.publisher)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325, in
__get__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
value = obj.__dict__[self.__name__] = self.__get(obj)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in publisher
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.Publisher(self._ensure_connected())
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in
ensure_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval_start, interval_step, interval_max, callback)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in
retry_over_time
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return fun(*args, **kwargs)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in
connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._connection = self._establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in
_establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) conn
= self.transport.establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in
establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) conn
= self.Connection(**opts)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in
__init__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._qpid_conn = establish(**self.connection_options)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112,
in establish
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn.open(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"<string>", line 6, in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323,
in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self.attach(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"<string>", line 6, in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 343,
in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
raise Timeout("Connection attach timed out")
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater) is starting.
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader -> celery.loaders.app.AppLoader
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service status I
can see that celery.bet fails to status. if I restart the server it and
immediately check the status it says running but checking the status
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ - ...
__
Post by Rick Langston
- _
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: Configuration
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . maxinterval
->
Post by Rick Langston
now (0s)
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your Katello
instance and ensure everything under "Backend System Status" says "OK"
with no further message.
If there are pulp errors, a possible quick fix is to ensure qpidd is
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up after
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are some other
possible things we can check, but I would be curious first about the
backend system status output.
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With a
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes normally
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data task that
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Chris Duryee
2016-08-01 17:19:17 UTC
Permalink
Post by Chris Duryee
Post by Rick Langston
Thank you for help on this
yes here is the actual capture
Thanks for the capture of the logs.
What is the output of systemctl status qpidd.service? I am curious if
only celerybeat is unable to connect, or if there is some other issue.
one more thing to check, as root please run:

qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
--ssl-key /etc/pki/katello/private/java-client.key -b
"amqps://$(hostname -f):5671" -q

If qpid is accepting connections, it should output some stuff about queues.
Post by Chris Duryee
If you want to restart qpidd, the best way is to just do a
katello-service stop. A number of other services depend on qpid, so it's
easiest to just down everything, ensure qpidd is stopped, then bring it
all back up via the script.
Post by Rick Langston
command line output
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s ago
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started Pulp's
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting Pulp's
Celerybeat...
beat: Starting...
pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
pulp.server.db.connection:INFO: Attempting to connect to localhost:27017
pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat v3.1.11
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ - ... __
- _
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: Configuration ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . maxinterval ->
now (0s)
/var/log/messages output
==> /var/log/messages <==
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) beat
Timeout('Connection attach timed out',)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat.start()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval = self.scheduler.tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line
265, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) ret
= self.call_tick(self, celerybeat_name)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line
230, in call_tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) ret
= super(Scheduler, self).tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
next_time_to_run = self.maybe_due(entry, self.publisher)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325, in
__get__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
value = obj.__dict__[self.__name__] = self.__get(obj)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in publisher
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.Publisher(self._ensure_connected())
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in
ensure_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval_start, interval_step, interval_max, callback)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in
retry_over_time
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return fun(*args, **kwargs)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in connect
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in
connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._connection = self._establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in
_establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) conn
= self.transport.establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600, in
establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) conn
= self.Connection(**opts)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261, in
__init__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._qpid_conn = establish(**self.connection_options)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 112,
in establish
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn.open(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"<string>", line 6, in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 323,
in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self.attach(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"<string>", line 6, in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) File
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 343,
in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
raise Timeout("Connection attach timed out")
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater) is starting.
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader -> celery.loaders.app.AppLoader
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service status I
can see that celery.bet fails to status. if I restart the server it and
immediately check the status it says running but checking the status
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ - ...
__
Post by Rick Langston
- _
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: Configuration
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . maxinterval
->
Post by Rick Langston
now (0s)
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your Katello
instance and ensure everything under "Backend System Status" says "OK"
with no further message.
If there are pulp errors, a possible quick fix is to ensure qpidd is
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up after
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are some other
possible things we can check, but I would be curious first about the
backend system status output.
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With a
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes normally
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data task that
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Rick Langston
2016-08-01 20:14:18 UTC
Permalink
Strange, If I restart the service "/bin/systemctl status
pulp_celerybeat.service" it will eventually start. Any ideas?
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Thank you for help on this
yes here is the actual capture
Thanks for the capture of the logs.
What is the output of systemctl status qpidd.service? I am curious if
only celerybeat is unable to connect, or if there is some other issue.
qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
--ssl-key /etc/pki/katello/private/java-client.key -b
"amqps://$(hostname -f):5671" -q
If qpid is accepting connections, it should output some stuff about queues.
Post by Chris Duryee
If you want to restart qpidd, the best way is to just do a
katello-service stop. A number of other services depend on qpid, so it's
easiest to just down everything, ensure qpidd is stopped, then bring it
all back up via the script.
Post by Rick Langston
command line output
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s ago
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started Pulp's
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting Pulp's
Celerybeat...
beat: Starting...
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat
v3.1.11
Post by Chris Duryee
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ - ...
__
Post by Chris Duryee
Post by Rick Langston
- _
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: Configuration
->
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . maxinterval
->
Post by Chris Duryee
Post by Rick Langston
now (0s)
/var/log/messages output
==> /var/log/messages <==
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) beat
Timeout('Connection attach timed out',)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat.start()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval = self.scheduler.tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line
265, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= self.call_tick(self, celerybeat_name)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line
230, in call_tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= super(Scheduler, self).tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
next_time_to_run = self.maybe_due(entry, self.publisher)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325,
in
Post by Chris Duryee
Post by Rick Langston
__get__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
value = obj.__dict__[self.__name__] = self.__get(obj)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
publisher
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.Publisher(self._ensure_connected())
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in
ensure_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval_start, interval_step, interval_max, callback)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246,
in
Post by Chris Duryee
Post by Rick Langston
retry_over_time
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return fun(*args, **kwargs)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in
connect
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in
connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._connection = self._establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in
_establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.transport.establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600,
in
Post by Chris Duryee
Post by Rick Langston
establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.Connection(**opts)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261,
in
Post by Chris Duryee
Post by Rick Langston
__init__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._qpid_conn = establish(**self.connection_options)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
112,
Post by Chris Duryee
Post by Rick Langston
in establish
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn.open(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
323,
Post by Chris Duryee
Post by Rick Langston
in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self.attach(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
343,
Post by Chris Duryee
Post by Rick Langston
in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
raise Timeout("Connection attach timed out")
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater) is
starting.
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader ->
celery.loaders.app.AppLoader
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service
status I
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
can see that celery.bet fails to status. if I restart the server it
and
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
immediately check the status it says running but checking the status
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
__
Post by Rick Langston
- _
Configuration
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler
->
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
now (0s)
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112,
in
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
start_scheduler
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your
Katello
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
instance and ensure everything under "Backend System Status" says
"OK"
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
with no further message.
If there are pulp errors, a possible quick fix is to ensure qpidd is
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up after
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are some
other
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
possible things we can check, but I would be curious first about the
backend system status output.
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With a
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes
normally
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data task
that
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Chris Duryee
2016-08-02 12:30:03 UTC
Permalink
Post by Rick Langston
Strange, If I restart the service "/bin/systemctl status
pulp_celerybeat.service" it will eventually start. Any ideas?
I'm glad it eventually restarts:) that is strange though that fails at
first but eventually works.

Can you send the output of foreman-debug? This will create an an archive
with system information and log files that we can peruse to debug further.
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Thank you for help on this
yes here is the actual capture
Thanks for the capture of the logs.
What is the output of systemctl status qpidd.service? I am curious if
only celerybeat is unable to connect, or if there is some other issue.
qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
--ssl-key /etc/pki/katello/private/java-client.key -b
"amqps://$(hostname -f):5671" -q
If qpid is accepting connections, it should output some stuff about queues.
Post by Chris Duryee
If you want to restart qpidd, the best way is to just do a
katello-service stop. A number of other services depend on qpid, so it's
easiest to just down everything, ensure qpidd is stopped, then bring it
all back up via the script.
Post by Rick Langston
command line output
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s ago
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started Pulp's
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting Pulp's
Celerybeat...
beat: Starting...
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.db.connection:INFO: Write concern for Mongo connection: {}
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat
v3.1.11
Post by Chris Duryee
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ - ...
__
Post by Chris Duryee
Post by Rick Langston
- _
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: Configuration
->
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . maxinterval
->
Post by Chris Duryee
Post by Rick Langston
now (0s)
/var/log/messages output
==> /var/log/messages <==
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168) beat
Timeout('Connection attach timed out',)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat.start()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval = self.scheduler.tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line
265, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= self.call_tick(self, celerybeat_name)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py", line
230, in call_tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= super(Scheduler, self).tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
next_time_to_run = self.maybe_due(entry, self.publisher)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325,
in
Post by Chris Duryee
Post by Rick Langston
__get__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
value = obj.__dict__[self.__name__] = self.__get(obj)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
publisher
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.Publisher(self._ensure_connected())
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in
ensure_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval_start, interval_step, interval_max, callback)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246,
in
Post by Chris Duryee
Post by Rick Langston
retry_over_time
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return fun(*args, **kwargs)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in
connect
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in
connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._connection = self._establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in
_establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.transport.establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1600,
in
Post by Chris Duryee
Post by Rick Langston
establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.Connection(**opts)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line 1261,
in
Post by Chris Duryee
Post by Rick Langston
__init__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._qpid_conn = establish(**self.connection_options)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
112,
Post by Chris Duryee
Post by Rick Langston
in establish
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn.open(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
323,
Post by Chris Duryee
Post by Rick Langston
in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self.attach(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
343,
Post by Chris Duryee
Post by Rick Langston
in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
raise Timeout("Connection attach timed out")
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater) is
starting.
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader ->
celery.loaders.app.AppLoader
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service
status I
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
can see that celery.bet fails to status. if I restart the server it
and
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
immediately check the status it says running but checking the status
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
__
Post by Rick Langston
- _
Configuration
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler
->
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
now (0s)
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112,
in
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
start_scheduler
Aug 1 07:25:16 dscaprv01 pulp: celery.beat:CRITICAL: (28691-76416)
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your
Katello
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
instance and ensure everything under "Backend System Status" says
"OK"
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
with no further message.
If there are pulp errors, a possible quick fix is to ensure qpidd is
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up after
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are some
other
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
possible things we can check, but I would be curious first about the
backend system status output.
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With a
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes
normally
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data task
that
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Rick Langston
2016-08-02 14:23:40 UTC
Permalink
and it will eventually die as it did last night.
Where would I send the debug file?
Post by Rick Langston
Strange, If I restart the service "/bin/systemctl status
pulp_celerybeat.service" it will eventually start. Any ideas?
I'm glad it eventually restarts:) that is strange though that fails at
first but eventually works.
Can you send the output of foreman-debug? This will create an an archive
with system information and log files that we can peruse to debug further.
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Thank you for help on this
yes here is the actual capture
Thanks for the capture of the logs.
What is the output of systemctl status qpidd.service? I am curious if
only celerybeat is unable to connect, or if there is some other issue.
qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
--ssl-key /etc/pki/katello/private/java-client.key -b
"amqps://$(hostname -f):5671" -q
If qpid is accepting connections, it should output some stuff about queues.
Post by Chris Duryee
If you want to restart qpidd, the best way is to just do a
katello-service stop. A number of other services depend on qpid, so
it's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
easiest to just down everything, ensure qpidd is stopped, then bring
it
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
all back up via the script.
Post by Rick Langston
command line output
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s ago
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started Pulp's
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting
Pulp's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Celerybeat...
beat: Starting...
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
{}
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat
v3.1.11
Post by Chris Duryee
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Rick Langston
Post by Chris Duryee
__
Post by Chris Duryee
Post by Rick Langston
- _
Configuration
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
now (0s)
/var/log/messages output
==> /var/log/messages <==
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat.start()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval = self.scheduler.tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
265, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= self.call_tick(self, celerybeat_name)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
230, in call_tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= super(Scheduler, self).tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
next_time_to_run = self.maybe_due(entry, self.publisher)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325,
in
Post by Chris Duryee
Post by Rick Langston
__get__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
value = obj.__dict__[self.__name__] = self.__get(obj)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
publisher
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.Publisher(self._ensure_connected())
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in
ensure_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval_start, interval_step, interval_max, callback)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246,
in
Post by Chris Duryee
Post by Rick Langston
retry_over_time
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return fun(*args, **kwargs)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in
connect
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in
connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._connection = self._establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in
_establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.transport.establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1600,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.Connection(**opts)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1261,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
__init__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._qpid_conn = establish(**self.connection_options)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
112,
Post by Chris Duryee
Post by Rick Langston
in establish
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn.open(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
323,
Post by Chris Duryee
Post by Rick Langston
in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self.attach(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
343,
Post by Chris Duryee
Post by Rick Langston
in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
raise Timeout("Connection attach timed out")
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater) is
starting.
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader ->
celery.loaders.app.AppLoader
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service
status I
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
can see that celery.bet fails to status. if I restart the server it
and
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
immediately check the status it says running but checking the
status
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
ago
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
__
Post by Rick Langston
- _
Configuration
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
scheduler
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
now (0s)
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
(28691-76416)
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112,
in
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
start_scheduler
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your
Katello
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
instance and ensure everything under "Backend System Status" says
"OK"
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
with no further message.
If there are pulp errors, a possible quick fix is to ensure qpidd
is
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up
after
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are some
other
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
possible things we can check, but I would be curious first about
the
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
backend system status output.
Post by Rick Langston
On Monday, August 1, 2016 at 7:12:26 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With
a
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes
normally
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data task
that
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Chris Duryee
2016-08-02 15:12:37 UTC
Permalink
Post by Rick Langston
and it will eventually die as it did last night.
Where would I send the debug file?
I guess the easiest way is to just email me direct with the attachment,
but we can keep the discussion on-list.
Post by Rick Langston
Post by Rick Langston
Strange, If I restart the service "/bin/systemctl status
pulp_celerybeat.service" it will eventually start. Any ideas?
I'm glad it eventually restarts:) that is strange though that fails at
first but eventually works.
Can you send the output of foreman-debug? This will create an an archive
with system information and log files that we can peruse to debug further.
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Thank you for help on this
yes here is the actual capture
Thanks for the capture of the logs.
What is the output of systemctl status qpidd.service? I am curious if
only celerybeat is unable to connect, or if there is some other issue.
qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
--ssl-key /etc/pki/katello/private/java-client.key -b
"amqps://$(hostname -f):5671" -q
If qpid is accepting connections, it should output some stuff about queues.
Post by Chris Duryee
If you want to restart qpidd, the best way is to just do a
katello-service stop. A number of other services depend on qpid, so
it's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
easiest to just down everything, ensure qpidd is stopped, then bring
it
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
all back up via the script.
Post by Rick Langston
command line output
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s ago
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started Pulp's
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting
Pulp's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Celerybeat...
beat: Starting...
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
{}
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s ago
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery beat
v3.1.11
Post by Chris Duryee
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Rick Langston
Post by Chris Duryee
__
Post by Chris Duryee
Post by Rick Langston
- _
Configuration
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader ->
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . scheduler
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile ->
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
now (0s)
/var/log/messages output
==> /var/log/messages <==
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112, in
start_scheduler
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat.start()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in start
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval = self.scheduler.tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
265, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= self.call_tick(self, celerybeat_name)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
230, in call_tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= super(Scheduler, self).tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
next_time_to_run = self.maybe_due(entry, self.publisher)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 325,
in
Post by Chris Duryee
Post by Rick Langston
__get__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
value = obj.__dict__[self.__name__] = self.__get(obj)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
publisher
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.Publisher(self._ensure_connected())
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369, in
ensure_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval_start, interval_step, interval_max, callback)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246,
in
Post by Chris Duryee
Post by Rick Langston
retry_over_time
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return fun(*args, **kwargs)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237, in
connect
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in
connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._connection = self._establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in
_establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.transport.establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1600,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.Connection(**opts)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1261,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
__init__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._qpid_conn = establish(**self.connection_options)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
112,
Post by Chris Duryee
Post by Rick Langston
in establish
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn.open(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
323,
Post by Chris Duryee
Post by Rick Langston
in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self.attach(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line
343,
Post by Chris Duryee
Post by Rick Langston
in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
raise Timeout("Connection attach timed out")
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater) is
starting.
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader ->
celery.loaders.app.AppLoader
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service
status I
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
can see that celery.bet fails to status. if I restart the server it
and
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
immediately check the status it says running but checking the
status
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
ago
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
__
Post by Rick Langston
- _
Configuration
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
scheduler
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
now (0s)
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
(28691-76416)
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112,
in
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
start_scheduler
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your
Katello
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
instance and ensure everything under "Backend System Status" says
"OK"
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
with no further message.
If there are pulp errors, a possible quick fix is to ensure qpidd
is
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up
after
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are some
other
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
possible things we can check, but I would be curious first about
the
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
backend system status output.
Post by Rick Langston
On Monday, August 1, 2016 at 7:12:26 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version. With
a
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes
normally
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data task
that
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Rick Langston
2016-08-03 11:41:14 UTC
Permalink
Things seem to stem from pulp and a connection

[***@servername ~]# /bin/systemctl status pulp_celerybeat.service
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Wed 2016-08-03 06:32:14 CDT; 1s ago
Process: 5388 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5388 (code=exited, status=0/SUCCESS)

Aug 03 06:32:13 servername.corp.company.net pulp[5388]:
celery.beat:CRITICAL: (5388-64256) Timeout: Connection attach timed out
Aug 03 06:32:13 servername.corp.company.net celery[5388]: celery beat
v3.1.11 (Cipater) is starting.
Aug 03 06:32:13 servername.corp.company.net celery[5388]: __ - ... __
- _
Aug 03 06:32:13 servername.corp.company.net celery[5388]: Configuration ->
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . broker ->
qpid://servername.corp.company.net:5671//
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . loader ->
celery.loaders.app.AppLoader
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . logfile ->
[stderr]@%INFO
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . maxinterval ->
now (0s)
Aug 03 06:32:14 servername.corp.company.net pulp[5388]:
qpid.messaging:DEBUG: poll(1 fds, timeout=4000.0)
Post by Chris Duryee
Post by Rick Langston
and it will eventually die as it did last night.
Where would I send the debug file?
I guess the easiest way is to just email me direct with the attachment,
but we can keep the discussion on-list.
Post by Rick Langston
Post by Rick Langston
Strange, If I restart the service "/bin/systemctl status
pulp_celerybeat.service" it will eventually start. Any ideas?
I'm glad it eventually restarts:) that is strange though that fails at
first but eventually works.
Can you send the output of foreman-debug? This will create an an
archive
Post by Rick Langston
with system information and log files that we can peruse to debug
further.
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Thank you for help on this
yes here is the actual capture
Thanks for the capture of the logs.
What is the output of systemctl status qpidd.service? I am curious
if
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
only celerybeat is unable to connect, or if there is some other
issue.
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
--ssl-key /etc/pki/katello/private/java-client.key -b
"amqps://$(hostname -f):5671" -q
If qpid is accepting connections, it should output some stuff about queues.
Post by Chris Duryee
If you want to restart qpidd, the best way is to just do a
katello-service stop. A number of other services depend on qpid, so
it's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
easiest to just down everything, ensure qpidd is stopped, then bring
it
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
all back up via the script.
Post by Rick Langston
command line output
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s
ago
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started
Pulp's
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting
Pulp's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Celerybeat...
beat: Starting...
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
{}
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
ago
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
beat
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
v3.1.11
Post by Chris Duryee
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Rick Langston
Post by Chris Duryee
__
Post by Chris Duryee
Post by Rick Langston
- _
Configuration
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
scheduler
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
now (0s)
/var/log/messages output
==> /var/log/messages <==
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
start_scheduler
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat.start()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in
start
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval = self.scheduler.tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
265, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= self.call_tick(self, celerybeat_name)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
230, in call_tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= super(Scheduler, self).tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in
tick
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
next_time_to_run = self.maybe_due(entry, self.publisher)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
325,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
__get__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
value = obj.__dict__[self.__name__] = self.__get(obj)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
publisher
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.Publisher(self._ensure_connected())
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
ensure_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval_start, interval_step, interval_max, callback)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
246,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
retry_over_time
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return fun(*args, **kwargs)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
connect
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._connection = self._establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
_establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.transport.establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1600,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.Connection(**opts)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1261,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
__init__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._qpid_conn = establish(**self.connection_options)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
112,
Post by Chris Duryee
Post by Rick Langston
in establish
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn.open(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
323,
Post by Chris Duryee
Post by Rick Langston
in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self.attach(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
343,
Post by Chris Duryee
Post by Rick Langston
in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
raise Timeout("Connection attach timed out")
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater) is
starting.
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader ->
celery.loaders.app.AppLoader
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service
status I
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
can see that celery.bet fails to status. if I restart the server
it
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
and
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
immediately check the status it says running but checking the
status
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded
(/usr/lib/systemd/system/pulp_celerybeat.service;
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
ago
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
__
Post by Rick Langston
- _
Configuration
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
scheduler
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
logfile
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
now (0s)
On Monday, August 1, 2016 at 8:22:46 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
(28691-76416)
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line
112,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
start_scheduler
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your
Katello
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
instance and ensure everything under "Backend System Status"
says
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
"OK"
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
with no further message.
If there are pulp errors, a possible quick fix is to ensure
qpidd
Post by Rick Langston
is
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up
after
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are
some
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
other
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
possible things we can check, but I would be curious first about
the
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
backend system status output.
Post by Rick Langston
On Monday, August 1, 2016 at 7:12:26 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version.
With
Post by Rick Langston
a
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes
normally
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data
task
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
that
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Chris Duryee
2016-08-03 15:41:45 UTC
Permalink
Post by Rick Langston
Things seem to stem from pulp and a connection
cool, this narrows it down some:)

Let's try this to see if it captures the issue:

# stop everything
katello-service stop

# start mongodb
systemctl start mongod

# become qpidd user
sudo su - qpidd -s /bin/bash (to become qpid user)

#start qpid with trace output
/usr/sbin/qpidd --config /etc/qpid/qpidd.conf -t 2>&1 | tee /tmp/qpid.out

in another window:

systemctl start pulp_celerybeat
systemctl start pulp_resource_manager
systemctl start pulp_workers

<wait a minute or so>

then stop the three pulp services, ^C to exit qpidd, and do a
"katello-service start" to spin everything up again.

Hopefully this will reproduce the issue, and we'll have the full qpid
log in /tmp/qpid.out to see what happened. The log is at trace setting
and will be extremely verbose.
Post by Rick Langston
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Wed 2016-08-03 06:32:14 CDT; 1s ago
Process: 5388 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5388 (code=exited, status=0/SUCCESS)
celery.beat:CRITICAL: (5388-64256) Timeout: Connection attach timed out
Aug 03 06:32:13 servername.corp.company.net celery[5388]: celery beat
v3.1.11 (Cipater) is starting.
Aug 03 06:32:13 servername.corp.company.net celery[5388]: __ - ... __
- _
Aug 03 06:32:13 servername.corp.company.net celery[5388]: Configuration ->
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . broker ->
qpid://servername.corp.company.net:5671//
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . loader ->
celery.loaders.app.AppLoader
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . logfile ->
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . maxinterval ->
now (0s)
qpid.messaging:DEBUG: poll(1 fds, timeout=4000.0)
Post by Chris Duryee
Post by Rick Langston
and it will eventually die as it did last night.
Where would I send the debug file?
I guess the easiest way is to just email me direct with the attachment,
but we can keep the discussion on-list.
Post by Rick Langston
Post by Rick Langston
Strange, If I restart the service "/bin/systemctl status
pulp_celerybeat.service" it will eventually start. Any ideas?
I'm glad it eventually restarts:) that is strange though that fails at
first but eventually works.
Can you send the output of foreman-debug? This will create an an
archive
Post by Rick Langston
with system information and log files that we can peruse to debug
further.
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Thank you for help on this
yes here is the actual capture
Thanks for the capture of the logs.
What is the output of systemctl status qpidd.service? I am curious
if
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
only celerybeat is unable to connect, or if there is some other
issue.
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
--ssl-key /etc/pki/katello/private/java-client.key -b
"amqps://$(hostname -f):5671" -q
If qpid is accepting connections, it should output some stuff about queues.
Post by Chris Duryee
If you want to restart qpidd, the best way is to just do a
katello-service stop. A number of other services depend on qpid, so
it's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
easiest to just down everything, ensure qpidd is stopped, then bring
it
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
all back up via the script.
Post by Rick Langston
command line output
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s
ago
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started
Pulp's
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting
Pulp's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Celerybeat...
beat: Starting...
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
{}
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
ago
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
beat
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
v3.1.11
Post by Chris Duryee
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Rick Langston
Post by Chris Duryee
__
Post by Chris Duryee
Post by Rick Langston
- _
Configuration
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
scheduler
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . logfile
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
now (0s)
/var/log/messages output
==> /var/log/messages <==
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
start_scheduler
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
beat.start()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in
start
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval = self.scheduler.tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
265, in tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= self.call_tick(self, celerybeat_name)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
230, in call_tick
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
ret
Post by Chris Duryee
Post by Rick Langston
= super(Scheduler, self).tick()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in
tick
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
next_time_to_run = self.maybe_due(entry, self.publisher)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
325,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
__get__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
value = obj.__dict__[self.__name__] = self.__get(obj)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
publisher
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.Publisher(self._ensure_connected())
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
ensure_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
interval_start, interval_step, interval_max, callback)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
246,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
retry_over_time
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return fun(*args, **kwargs)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
connect
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
return self.connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._connection = self._establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
_establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.transport.establish_connection()
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1600,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
establish_connection
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn
Post by Chris Duryee
Post by Rick Langston
= self.Connection(**opts)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1261,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
__init__
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self._qpid_conn = establish(**self.connection_options)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
112,
Post by Chris Duryee
Post by Rick Langston
in establish
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
conn.open(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
323,
Post by Chris Duryee
Post by Rick Langston
in open
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
self.attach(timeout=timeout)
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
343,
Post by Chris Duryee
Post by Rick Langston
in attach
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
raise Timeout("Connection attach timed out")
Aug 1 08:45:03 dscaprv01 pulp: celery.beat:CRITICAL: (7070-59168)
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater) is
starting.
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader ->
celery.loaders.app.AppLoader
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service
status I
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
can see that celery.bet fails to status. if I restart the server
it
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
and
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
immediately check the status it says running but checking the
status
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded
(/usr/lib/systemd/system/pulp_celerybeat.service;
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
ago
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
__
Post by Rick Langston
- _
Configuration
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
scheduler
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
logfile
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
now (0s)
On Monday, August 1, 2016 at 8:22:46 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
(28691-76416)
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line
112,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
start_scheduler
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your
Katello
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
instance and ensure everything under "Backend System Status"
says
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
"OK"
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
with no further message.
If there are pulp errors, a possible quick fix is to ensure
qpidd
Post by Rick Langston
is
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up
after
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are
some
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
other
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
possible things we can check, but I would be curious first about
the
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
backend system status output.
Post by Rick Langston
On Monday, August 1, 2016 at 7:12:26 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version.
With
Post by Rick Langston
a
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes
normally
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data
task
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
that
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Rick Langston
2016-08-04 11:38:22 UTC
Permalink
Well this is strange. it looks as if the process has stabilized. I didn't
see anything in the capture that jumped out at me but will try and
normalize the build today and see if it continues to work.

Thank you for your help with this
Post by Chris Duryee
Post by Rick Langston
Things seem to stem from pulp and a connection
cool, this narrows it down some:)
# stop everything
katello-service stop
# start mongodb
systemctl start mongod
# become qpidd user
sudo su - qpidd -s /bin/bash (to become qpid user)
#start qpid with trace output
/usr/sbin/qpidd --config /etc/qpid/qpidd.conf -t 2>&1 | tee /tmp/qpid.out
systemctl start pulp_celerybeat
systemctl start pulp_resource_manager
systemctl start pulp_workers
<wait a minute or so>
then stop the three pulp services, ^C to exit qpidd, and do a
"katello-service start" to spin everything up again.
Hopefully this will reproduce the issue, and we'll have the full qpid
log in /tmp/qpid.out to see what happened. The log is at trace setting
and will be extremely verbose.
Post by Rick Langston
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Wed 2016-08-03 06:32:14 CDT; 1s ago
Process: 5388 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5388 (code=exited, status=0/SUCCESS)
celery.beat:CRITICAL: (5388-64256) Timeout: Connection attach timed out
Aug 03 06:32:13 servername.corp.company.net celery[5388]: celery beat
v3.1.11 (Cipater) is starting.
Aug 03 06:32:13 servername.corp.company.net celery[5388]: __ -
... __
Post by Rick Langston
- _
Aug 03 06:32:13 servername.corp.company.net celery[5388]: Configuration
->
Post by Rick Langston
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . broker ->
qpid://servername.corp.company.net:5671//
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . loader ->
celery.loaders.app.AppLoader
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . scheduler
->
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . logfile ->
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . maxinterval
->
Post by Rick Langston
now (0s)
qpid.messaging:DEBUG: poll(1 fds, timeout=4000.0)
Post by Chris Duryee
Post by Rick Langston
and it will eventually die as it did last night.
Where would I send the debug file?
I guess the easiest way is to just email me direct with the attachment,
but we can keep the discussion on-list.
Post by Rick Langston
Post by Rick Langston
Strange, If I restart the service "/bin/systemctl status
pulp_celerybeat.service" it will eventually start. Any ideas?
I'm glad it eventually restarts:) that is strange though that fails
at
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
first but eventually works.
Can you send the output of foreman-debug? This will create an an
archive
Post by Rick Langston
with system information and log files that we can peruse to debug
further.
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Thank you for help on this
yes here is the actual capture
Thanks for the capture of the logs.
What is the output of systemctl status qpidd.service? I am curious
if
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
only celerybeat is unable to connect, or if there is some other
issue.
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
--ssl-key /etc/pki/katello/private/java-client.key -b
"amqps://$(hostname -f):5671" -q
If qpid is accepting connections, it should output some stuff about queues.
Post by Chris Duryee
If you want to restart qpidd, the best way is to just do a
katello-service stop. A number of other services depend on qpid,
so
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
it's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
easiest to just down everything, ensure qpidd is stopped, then
bring
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
it
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
all back up via the script.
Post by Rick Langston
command line output
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded
(/usr/lib/systemd/system/pulp_celerybeat.service;
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s
ago
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started
Pulp's
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting
Pulp's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Celerybeat...
beat: Starting...
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.db.connection:INFO: Write concern for Mongo
{}
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded
(/usr/lib/systemd/system/pulp_celerybeat.service;
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
ago
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
beat
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
v3.1.11
Post by Chris Duryee
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Rick Langston
Post by Chris Duryee
__
Post by Chris Duryee
Post by Rick Langston
- _
Configuration
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
scheduler
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
logfile
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
now (0s)
/var/log/messages output
==> /var/log/messages <==
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
(7070-59168)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
start_scheduler
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
beat.start()
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in
start
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
interval = self.scheduler.tick()
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
265, in tick
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
ret
Post by Chris Duryee
Post by Rick Langston
= self.call_tick(self, celerybeat_name)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
230, in call_tick
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
ret
Post by Chris Duryee
Post by Rick Langston
= super(Scheduler, self).tick()
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in
tick
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
next_time_to_run = self.maybe_due(entry, self.publisher)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
325,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
__get__
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
value = obj.__dict__[self.__name__] = self.__get(obj)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
publisher
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
return self.Publisher(self._ensure_connected())
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
ensure_connection
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
interval_start, interval_step, interval_max, callback)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
246,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
retry_over_time
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
return fun(*args, **kwargs)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
connect
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
return self.connection
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
connection
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
self._connection = self._establish_connection()
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
_establish_connection
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
conn
Post by Chris Duryee
Post by Rick Langston
= self.transport.establish_connection()
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1600,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
establish_connection
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
conn
Post by Chris Duryee
Post by Rick Langston
= self.Connection(**opts)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1261,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
__init__
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
self._qpid_conn = establish(**self.connection_options)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
112,
Post by Chris Duryee
Post by Rick Langston
in establish
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
conn.open(timeout=timeout)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in open
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
323,
Post by Chris Duryee
Post by Rick Langston
in open
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
self.attach(timeout=timeout)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in attach
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
343,
Post by Chris Duryee
Post by Rick Langston
in attach
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
raise Timeout("Connection attach timed out")
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater)
is
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
starting.
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader ->
celery.loaders.app.AppLoader
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
On Monday, August 1, 2016 at 8:43:45 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service
status I
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
can see that celery.bet fails to status. if I restart the
server
Post by Rick Langston
Post by Chris Duryee
it
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
and
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
immediately check the status it says running but checking the
status
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded
(/usr/lib/systemd/system/pulp_celerybeat.service;
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT;
18s
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
ago
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __
-
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
...
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
__
Post by Rick Langston
- _
Configuration
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
broker
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
loader
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
scheduler
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
logfile
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
now (0s)
On Monday, August 1, 2016 at 8:22:46 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
(28691-76416)
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line
112,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
start_scheduler
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your
Katello
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
instance and ensure everything under "Backend System Status"
says
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
"OK"
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
with no further message.
If there are pulp errors, a possible quick fix is to ensure
qpidd
Post by Rick Langston
is
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up
after
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are
some
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
other
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
possible things we can check, but I would be curious first
about
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
the
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
backend system status output.
Post by Rick Langston
On Monday, August 1, 2016 at 7:12:26 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version.
With
Post by Rick Langston
a
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes
normally
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data
task
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
that
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Chris Duryee
2016-08-04 17:03:01 UTC
Permalink
Post by Rick Langston
Well this is strange. it looks as if the process has stabilized. I didn't
see anything in the capture that jumped out at me but will try and
normalize the build today and see if it continues to work.
Well, I am glad it works now at least:) If it becomes unstable again let
us know and we can look more.
Post by Rick Langston
Thank you for your help with this
No problem, thanks for debugging with us.
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Things seem to stem from pulp and a connection
cool, this narrows it down some:)
# stop everything
katello-service stop
# start mongodb
systemctl start mongod
# become qpidd user
sudo su - qpidd -s /bin/bash (to become qpid user)
#start qpid with trace output
/usr/sbin/qpidd --config /etc/qpid/qpidd.conf -t 2>&1 | tee /tmp/qpid.out
systemctl start pulp_celerybeat
systemctl start pulp_resource_manager
systemctl start pulp_workers
<wait a minute or so>
then stop the three pulp services, ^C to exit qpidd, and do a
"katello-service start" to spin everything up again.
Hopefully this will reproduce the issue, and we'll have the full qpid
log in /tmp/qpid.out to see what happened. The log is at trace setting
and will be extremely verbose.
Post by Rick Langston
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Wed 2016-08-03 06:32:14 CDT; 1s ago
Process: 5388 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5388 (code=exited, status=0/SUCCESS)
celery.beat:CRITICAL: (5388-64256) Timeout: Connection attach timed out
Aug 03 06:32:13 servername.corp.company.net celery[5388]: celery beat
v3.1.11 (Cipater) is starting.
Aug 03 06:32:13 servername.corp.company.net celery[5388]: __ -
... __
Post by Rick Langston
- _
Aug 03 06:32:13 servername.corp.company.net celery[5388]: Configuration
->
Post by Rick Langston
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . broker ->
qpid://servername.corp.company.net:5671//
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . loader ->
celery.loaders.app.AppLoader
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . scheduler
->
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . logfile ->
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . maxinterval
->
Post by Rick Langston
now (0s)
qpid.messaging:DEBUG: poll(1 fds, timeout=4000.0)
Post by Chris Duryee
Post by Rick Langston
and it will eventually die as it did last night.
Where would I send the debug file?
I guess the easiest way is to just email me direct with the attachment,
but we can keep the discussion on-list.
Post by Rick Langston
Post by Rick Langston
Strange, If I restart the service "/bin/systemctl status
pulp_celerybeat.service" it will eventually start. Any ideas?
I'm glad it eventually restarts:) that is strange though that fails
at
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
first but eventually works.
Can you send the output of foreman-debug? This will create an an
archive
Post by Rick Langston
with system information and log files that we can peruse to debug
further.
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Thank you for help on this
yes here is the actual capture
Thanks for the capture of the logs.
What is the output of systemctl status qpidd.service? I am curious
if
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
only celerybeat is unable to connect, or if there is some other
issue.
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt
--ssl-key /etc/pki/katello/private/java-client.key -b
"amqps://$(hostname -f):5671" -q
If qpid is accepting connections, it should output some stuff about queues.
Post by Chris Duryee
If you want to restart qpidd, the best way is to just do a
katello-service stop. A number of other services depend on qpid,
so
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
it's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
easiest to just down everything, ensure qpidd is stopped, then
bring
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
it
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
all back up via the script.
Post by Rick Langston
command line output
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded
(/usr/lib/systemd/system/pulp_celerybeat.service;
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-01 08:34:20 CDT; 7s
ago
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Main PID: 5887 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─5887 /usr/bin/python /usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Started
Pulp's
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Celerybeat.
Aug 01 08:34:20 dscaprv01.corp.acxiom.net systemd[1]: Starting
Pulp's
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Celerybeat...
beat: Starting...
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler:INFO: Worker Timeout Monitor Started
pulp.server.db.connection:INFO: Attempting to connect to
localhost:27017
Post by Chris Duryee
Post by Rick Langston
pulp.server.db.connection:INFO: Write concern for Mongo
{}
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded
(/usr/lib/systemd/system/pulp_celerybeat.service;
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT; 18s
ago
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
beat
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
v3.1.11
Post by Chris Duryee
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __ -
...
Post by Rick Langston
Post by Chris Duryee
__
Post by Chris Duryee
Post by Rick Langston
- _
Configuration
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . broker
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: . loader
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
scheduler
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
logfile
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
now (0s)
/var/log/messages output
==> /var/log/messages <==
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
(7070-59168)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line 112,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
start_scheduler
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
beat.start()
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 462, in
start
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
interval = self.scheduler.tick()
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
265, in tick
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
ret
Post by Chris Duryee
Post by Rick Langston
= self.call_tick(self, celerybeat_name)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
"/usr/lib/python2.7/site-packages/pulp/server/async/scheduler.py",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
line
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
230, in call_tick
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
ret
Post by Chris Duryee
Post by Rick Langston
= super(Scheduler, self).tick()
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 220, in
tick
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
next_time_to_run = self.maybe_due(entry, self.publisher)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
325,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
__get__
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
value = obj.__dict__[self.__name__] = self.__get(obj)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 342, in
publisher
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
return self.Publisher(self._ensure_connected())
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/beat.py", line 326, in
_ensure_connected
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
_error_handler, self.app.conf.BROKER_CONNECTION_MAX_RETRIES
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 369,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
ensure_connection
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
interval_start, interval_step, interval_max, callback)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line
246,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
retry_over_time
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
return fun(*args, **kwargs)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 237,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
connect
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
return self.connection
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 741,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
connection
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
self._connection = self._establish_connection()
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/connection.py", line 696,
in
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
_establish_connection
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
conn
Post by Chris Duryee
Post by Rick Langston
= self.transport.establish_connection()
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1600,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
establish_connection
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
conn
Post by Chris Duryee
Post by Rick Langston
= self.Connection(**opts)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/kombu/transport/qpid.py", line
1261,
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
__init__
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
self._qpid_conn = establish(**self.connection_options)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
112,
Post by Chris Duryee
Post by Rick Langston
in establish
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
conn.open(timeout=timeout)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in open
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
323,
Post by Chris Duryee
Post by Rick Langston
in open
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
self.attach(timeout=timeout)
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"<string>", line 6, in attach
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
File
Post by Chris Duryee
Post by Rick Langston
"/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py",
line
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
343,
Post by Chris Duryee
Post by Rick Langston
in attach
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
raise Timeout("Connection attach timed out")
(7070-59168)
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Connection attach timed out
Aug 1 08:45:03 dscaprv01 celery: celery beat v3.1.11 (Cipater)
is
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
starting.
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: __ - ... __ - _
Aug 1 08:45:03 dscaprv01 celery: Configuration ->
Aug 1 08:45:03 dscaprv01 celery: . broker ->
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 1 08:45:03 dscaprv01 celery: . loader ->
celery.loaders.app.AppLoader
Post by Chris Duryee
Post by Rick Langston
Aug 1 08:45:03 dscaprv01 celery: . scheduler ->
pulp.server.async.scheduler.Scheduler
Aug 1 08:45:03 dscaprv01 celery: . maxinterval -> now (0s)
On Monday, August 1, 2016 at 8:43:45 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
The backend service all say ok but when a run a katelli-service
status I
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
can see that celery.bet fails to status. if I restart the
server
Post by Rick Langston
Post by Chris Duryee
it
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
and
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
immediately check the status it says running but checking the
status
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
again
Post by Rick Langston
shows it timed out.
Is this the sequence of events?
* service pulp_celerybeat start (outputs success)
* service pulp_celerybeat status (outputs success)
* wait some number of seconds
* service pulp_celerybeat status (outputs error)
Post by Rick Langston
No memory errros noted
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded
(/usr/lib/systemd/system/pulp_celerybeat.service;
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
enabled; vendor preset: disabled)
Active: inactive (dead) since Mon 2016-08-01 08:34:31 CDT;
18s
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
ago
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Process: 5887 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5887 (code=exited, status=0/SUCCESS)
(5887-79264) raise Timeout("Connection attach timed out")
(5887-79264) Timeout: Connection attach timed out
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: celery
beat
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
v3.1.11
Post by Rick Langston
(Cipater) is starting.
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: __
-
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
...
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
__
Post by Rick Langston
- _
Configuration
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
broker
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
qpid://dscaprv01.corp.acxiom.net:5671//
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
loader
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
celery.loaders.app.AppLoader
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
scheduler
Post by Rick Langston
Post by Chris Duryee
->
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
logfile
Post by Rick Langston
->
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Aug 01 08:34:30 dscaprv01.corp.acxiom.net celery[5887]: .
maxinterval
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
now (0s)
On Monday, August 1, 2016 at 8:22:46 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
I do see this issue in messages but not sure if its related
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
beat
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Timeout('Connection attach timed out',)
(28691-76416)
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
File
Post by Rick Langston
"/usr/lib/python2.7/site-packages/celery/apps/beat.py", line
112,
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
in
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
start_scheduler
(28691-76416)
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
beat.start()
That is the likely culprit:)
Next time your task hangs, check in the "/about" page on your
Katello
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
instance and ensure everything under "Backend System Status"
says
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
"OK"
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
with no further message.
If there are pulp errors, a possible quick fix is to ensure
qpidd
Post by Rick Langston
is
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
still running, then restart pulp_workers, pulp_celerybeat and
pulp_resource_manager. I suspect your task will get picked up
after
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
that.
Post by Rick Langston
Post by Chris Duryee
Also, please check dmesg for out-of-memory errors. There are
some
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
other
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
possible things we can check, but I would be curious first
about
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
the
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
backend system status output.
Post by Rick Langston
On Monday, August 1, 2016 at 7:12:26 AM UTC-5, Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Hello
I seem to be having some odd behavior with this version.
With
Post by Rick Langston
a
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
fresh
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
install on centos 7 I have setup a product which completes
normally
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
but
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
when I discover a repo and save them i get these meta data
task
Post by Rick Langston
Post by Rick Langston
Post by Chris Duryee
that
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
seem
Post by Rick Langston
to just wait forever. Any ideas what can be the culprit
Are there any related errors in /var/log/messages?
Post by Rick Langston
Actions::Pulp::Repository::DistributorPublish
State: waiting for Pulp to start the task
{"pulp_id"=>"test-centos-6_updates_x86_64",
"distributor_type_id"=>"yum_distributor",
"source_pulp_id"=>nil,
"dependency"=>nil,
"remote_user"=>"admin",
"remote_cp_user"=>"admin",
"locale"=>"en"}
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.publish.publish",
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"_href"=>"/pulp/api/v2/tasks/a40815d5-9ba4-463a-8216-338cdcc4b1cc/",
Post by Rick Langston
Post by Chris Duryee
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
"task_id"=>"a40815d5-9ba4-463a-8216-338cdcc4b1cc",
"tags"=>
["pulp:repository:test-centos-6_updates_x86_64",
"pulp:action:publish"],
Post by Rick Langston
"finish_time"=>nil,
"_ns"=>"task_status",
"start_time"=>nil,
"traceback"=>nil,
"spawned_tasks"=>[],
"progress_report"=>{},
"queue"=>"None.dq",
"state"=>"waiting",
"worker_name"=>nil,
"result"=>nil,
"error"=>nil,
"_id"=>{"$oid"=>"579f32aa95c48c6a54257674"},
"id"=>"579f32aa95c48c6a54257674"}],
"poll_attempts"=>{"total"=>100, "failed"=>0}}
--
You received this message because you are subscribed to the Google Groups "Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+***@googlegroups.com.
To post to this group, send email to foreman-***@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.
Rick Langston
2016-08-05 10:48:39 UTC
Permalink
Well the rebuild came back with the same issue. This issue seems to be
tied to this hardware somehow as I can build using the same process in a VM
and everything is as it should be. Diags have been ran on the hardware
with no issues found. we also removed the external storage and mounted
everything local with no change.

Are there any tunes that we need to be aware of in regards to
CPU,disk,memory or open files sockets etc?

Thank you
Post by Rick Langston
Post by Rick Langston
Well this is strange. it looks as if the process has stabilized. I
didn't
Post by Rick Langston
see anything in the capture that jumped out at me but will try and
normalize the build today and see if it continues to work.
Well, I am glad it works now at least:) If it becomes unstable again let
us know and we can look more.
Post by Rick Langston
Thank you for your help with this
No problem, thanks for debugging with us.
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Things seem to stem from pulp and a connection
cool, this narrows it down some:)
# stop everything
katello-service stop
# start mongodb
systemctl start mongod
# become qpidd user
sudo su - qpidd -s /bin/bash (to become qpid user)
#start qpid with trace output
/usr/sbin/qpidd --config /etc/qpid/qpidd.conf -t 2>&1 | tee
/tmp/qpid.out
Post by Rick Langston
Post by Chris Duryee
systemctl start pulp_celerybeat
systemctl start pulp_resource_manager
systemctl start pulp_workers
<wait a minute or so>
then stop the three pulp services, ^C to exit qpidd, and do a
"katello-service start" to spin everything up again.
Hopefully this will reproduce the issue, and we'll have the full qpid
log in /tmp/qpid.out to see what happened. The log is at trace setting
and will be extremely verbose.
Post by Rick Langston
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service;
enabled; vendor preset: disabled)
Active: inactive (dead) since Wed 2016-08-03 06:32:14 CDT; 1s ago
Process: 5388 ExecStart=/usr/bin/celery beat
--app=pulp.server.async.celery_instance.celery
--scheduler=pulp.server.async.scheduler.Scheduler (code=exited,
status=0/SUCCESS)
Main PID: 5388 (code=exited, status=0/SUCCESS)
celery.beat:CRITICAL: (5388-64256) Timeout: Connection attach timed
out
Post by Rick Langston
Post by Chris Duryee
Post by Rick Langston
Aug 03 06:32:13 servername.corp.company.net celery[5388]: celery beat
v3.1.11 (Cipater) is starting.
Aug 03 06:32:13 servername.corp.company.net celery[5388]: __ -
... __
Post by Rick Langston
- _
Configuration
Post by Rick Langston
Post by Chris Duryee
->
Post by Rick Langston
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . broker ->
qpid://servername.corp.company.net:5671//
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . loader ->
celery.loaders.app.AppLoader
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . scheduler
->
Post by Rick Langston
pulp.server.async.scheduler.Scheduler
Aug 03 06:32:13 servername.corp.company.net celery[5388]: . logfile
->