Upstart log to file

This log file looks like it has all node messages for every run, as opposed to the normal roslaunch logs which create a separate file for each node each time roslaunch is called.

I wish there was a better log file to look at but at least its something. Please start posting anonymously - your entry will be published after you log in or create a new account. Asked: How can I get my cout output on the console with roslaunch. The third part is that messages written to the kernel ring buffer from user space actually look different from messages written from kernel space: they have a different facility. Now here's the clincher: by default, rsyslogd ignores messages with a non-kernel facility when it's reading messages from the kernel ring buffer.

Note that the rest of rsyslogd's config will treat all messages from the kernel ring buffer as having the kern facility, irrespective of the facility they had in the kernel ring buffer. Code can write to syslog by calling the glibc function syslog , described in man 3 syslog. Kernel space writes to this buffer by calling the kernel function printk , so it's sometimes called the printk buffer. Traditionally, klogd reads from one of these interfaces, then calls the glibc function syslog to copy them to the syslog.

There is the added caveat that all imklog output has the kern facility irrespective of the facility messages had in the kernel ring buffer. Ubuntu Community Ask! Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Learn more. Where are Upstart log messages on Ubuntu Ask Question. Asked 7 years, 9 months ago.

Like 0. Join the DZone community and get the full member experience. Join For Free. What does this look like in action? She sells seashells by the sea shore. The shells she sells are seashells I'm sure. What happens when we delete the file in between writes? I slowed the output down to one line per ten seconds, and then deleted the file right after one line had been flushed to disk. Great, that makes sense, but does it work as well when there is a high volume of data coming in?

Removing the sleep in the script, I let the log build up to about 2GB before doing anything. To simulate realistic rotation conditions, I simply invoked gzip on the log file to allow it to compress and then delete the original file. If the post-stop stanza exists, the post-stop process is spawned. The state is changed from post-stop to waiting. The stopped 7 event is emitted. Any jobs whose start on or stop on condition would be satisfied by this job being stopped are started or stopped respectively.

Note: this information is also available in upstart-events 7. As a general rule, you cannot rely upon the the order in which events will be emitted. Your system is dynamic and Upstart responds to changes as-and-when they occur for example hot-plug events. That said, most systems which use Upstart provide a number of "well-known" events which you can rely upon. For example on Ubuntu , these are documented in the upstart-events 7 man page, which is included within this document for convenience in appendix Ubuntu Well-Known Events ubuntu-specific.

Question: If event event-A is emitted, which job will run first? Answer: It is not possible to say, and indeed you should not make any assumptions about the order in which jobs with the same conditions run in. Would Upstart be happy with this?

Actually, yes it would! Upstart always handles stop on stanzas before handling start on stanzas. This means that this strange job would first be stopped if it's currently running , then it would be started. We can see what happens when we run this job more clearly when we increase the log priority to debug see Change the log-priority :.

Here is an example from the system log showing what happened in more detail. First the entries relating to starting the job:. Upstart guarantees that jobs which stop on a particular event are processed before jobs that start on the same event.

Assuming that job " A " is already running, if the " foo " event is emitted, Upstart will always stop job " A " before starting job " B ". A runlevel is a single-byte name for a particular system configuration. Runlevels for Debian and Ubuntu systems are generally as follows [34] :. Note that if this command is unable to determine the system runlevel, it may display simply " unknown ":.

For example, to make the system boot by default to single user mode, set:. If you want to change the default runlevel for a single boot, rather than making the change permanent by modify the rc-sysinit.

However, with Upstart, this file is no longer used it is supported by Upstart, but its use is deprecated. To obtain a better understanding of how jobs and events relate at startup and shutdown time, see Visualising Jobs and Events.

At boot, after the initramfs system has been run for setting up RAID, unlocking encrypted file system volumes, et cetera , Upstart will be given control. Note that in this section we assume the default runlevel is " 2 ". See Changing the Default Runlevel for further details. Upstart itself emits a single event called startup 7. This event triggers the rest of the system to initialize [33].

The most notable of these is the mountall job which mounts your disks and filesystems. The mountall 8 job in turn emits a number of events. These include local-filesystems 7 , virtual-filesystems 7 and all-swaps 7.

See upstart-events 7 for further details. The virtual-filesystems 7 event causes the udev job to start. The udev job causes the upstart-udev-bridge job to start.

After the last filesystem is mounted, mountall 8 will emit the filesystem event. Since the start on condition for the rc-sysinit job is:. Upstart will then start the rc-sysinit job. The rc-sysinit job calls the telinit command, passing it the runlevel to move to:. The telinit command emits the runlevel 7 event as:. Note that this is all the telinit command does — it runs no commands itself to change runlevel!

See Runlevels for further information on runlevels. Upstart will "die" when the system is powered off, but if it ever exits, that is a bug. Upstart never stops a job with no stop on condition. Ubuntu currently employs a hybrid system where core services are handled by Upstart, but additional services can be run in the legacy SystemV mode.

This may seem odd, but consider that there are thousands of packages available in Ubuntu via the Universe and Multiverse repositories and hundreds of services. To avoid having to change every package to work with Upstart, Upstart allows packages to utilize their existing SystemV and thus Debian-compatible scripts. Click "Shut Down Run the shutdown 8 command, for example:. Assuming the current runlevel is " 2 ", either of the actions above will cause Upstart to emit the runlevel 7 event like this:.

Click "Restart Run the shutdown 8 command specifying the " -r " option, for example:. Run the reboot 8 command:. Assuming the current runlevel is " 2 ", whichever command is run above will cause Upstart to emit the runlevel 7 event like this:.

When booting direct into single-user mode, the runlevel command will show:. Ubuntu provides a recovery mode in case your system experiences problems. This is handled by the friendly-recovery package. If you select a " recovery mode " option on the Grub menu.

As a result, this job has full control over the system and provides a friendly menu that allows users to check disks with fsck 8 , repair your package database and so on. This is a new phase introduced in Ubuntu A new job called failsafe has been introduced that checks to ensure the system has reached a particular state. If the expected state is not attained, the job reboots the system automatically. This section lists a number of job configuration file stanzas, giving example usage for each.

The reference for your specific version of Upstart will be available in the init 5 man page. Load specified AppArmor Mandatory Access Control system profile into the kernel prior to starting the job. The main job process as specified by exec or script will be confined to this profile. Quoted name and maybe contact details of author of this Job Configuration File. A new " cgroup " stanza is introduced that allows job processes to be run within the specified cgroup.

This allows the job to specify the control group all job processes will run in and optionally specify a setting for the particular cgroup. If only the cgroup controller such as memory , cpuset , blkio is specified, a job-specific cgroup will be created and the job processes placed in it.

The form of this cgroup is:. This variable is not an environment variable and is only valid within the context of the cgroup stanza. The NAME argument may contain any valid variable and can also contain forward slashes to run the job processes in a sub-cgroup. The stanza maybe specified multiple times. The last occurence will be used except in the scenario where each occurence specifies a different KEY in which case all the keys and values will be applied. Run all job processes in the default cpu cgroup controller group:.

For all versions of Upstart prior to v1. As of Upstart 1. If you are using Upstart 1. Using this option it is possible to set the default to none but still honour jobs that specify explicitly console log. If a User Job running in a pre-Upstart 1. Connects the job's standard input, standard output and standard error file descriptors to the console device. Identical to console output except that additionally it makes the job the owner of the console device.

This means it will receive certain signals from the kernel when special key combinations such as Control-C are pressed. Runs the job's processes with a working directory in the specified directory instead of the root of the filesystem. Runs the job's processes in a chroot 8 environment underneath the specified directory.

One line quoted description of Job Configuration File. Specifies the events the job configuration file generates directly or indirectly via a child process.

This stanza can be specified multiple times for each event emitted. This stanza can also use the following shell wildcard meta-characters to simplify the specification:. For example, upstart-udev-bridge can emit a large number of events. Rather than having to specify every possible event, since the form of the event names is consistent, a single emits stanza can be specified to cover all possible events:. Stanza that allows the specification of a single-line command to run.

Note that if this command-line contains any shell meta-characters, it will be passed through a shell prior to being executed. This ensures that shell redirection and variable expansion occur as expected.

Upstart will keep track of the process ID that it thinks belongs to a job. If a job has specified the instance stanza, Upstart will track the PIDs for each unique instance of that job. If you do not specify the expect stanza, Upstart will track the life cycle of the first PID that it executes in the exec or script stanzas.

However, most Unix services will "daemonize", meaning that they will create a new process using fork 2 which is a child of the initial process. Often services will "double fork" to ensure they have no association whatsoever with the initial process. Note that no services will fork more than twice initially since there is no additional benefit in doing so.

In this case, Upstart must have a way to track it, so you can use expect fork , or expect daemon which allows Upstart to use ptrace 2 to "count forks". To allow Upstart to determine the final process ID for a job, it needs to know how many times that process will call fork 2. Upstart itself cannot know the answer to this question since once a daemon is running, it could then fork a number of "worker" processes which could themselves fork any number of times.

Upstart cannot be expected to know which PID is the "master" in this case, considering it does not know if worker processes will be created at all, let alone how many times, or how many times the process will fork initially. This is achieved using the expect stanza. If your daemon has a "don't daemonize" or "run in the foreground" mode, then it's much simpler to use that and not run with fork following.

A final point: the expect stanza only applies to exec and script stanzas: it has no effect on pre-start and post-start. It's important to note that the " expect " stanza is thus being used for two different but complementary tasks:. Upstart will expect the process executed to call fork 2 exactly once.

In this case, expect fork cannot be used. See Daemon Behaviour. Upstart will expect the process executed to call fork 2 exactly twice. If the application you are attempting to create a Job Configuration File does not document how many times it forks, you can run it with a tool such as strace 1 which will allow you to count the number of forks.

The table below summarizes the behaviour resulting for every combination of expect stanza and number of fork 2 calls:. The start command will "hang" if you have misspecified the expect stanza by telling Upstart to expect more fork 2 calls than your application actually makes.

Run the initctl status command for your job. You will see something like:. Kill 1 the PID of your application. Re-run the initctl status command for your job. Correct the expect stanza specification in the job configuration file. If you have misspecified the expect stanza by telling Upstart to expect fewer fork 2 calls than your application actually makes, Upstart will be unable to manage it since it will be looking at the wrong PID. Find your jobs PID using ps 1. If you're struggling to find it, remember that the parent PID will always be " 1 ".

Export variables previously set with env to all events that result from this job. See for example Job Lifecycle. Sometimes you want to run the same job, but with different arguments. The variable that defines the unique instance of this job is defined with instance.

Let us start with a simple example which we will call " foo. Note that the entire job is the instance job: providing the instance stanza allows Upstart to make each running version of this job unique. We forgot to specify the particular value for the BAR variable which makes each instance unique.

Lets try again:. We tried to run another instance with the same instance name well, the same value of the BAR variable technically.

Good - Upstart is running two instances as expected. Notice the instance name in brackets after the job name in the initctl output above. That fails as Upstart needs to know which instance to stop and we didn't specify an instance value for the BAR instance variable. Rather than stopping each instance in turn, let's script it so that we can stop then all in one go:. In this way, Upstart will keep them all running with the specified arguments, and stop them if memcached is ever stopped.

You can do silly things like the following if you wish:. If you attempt to start a job with the instance stanza, but forget to provide the required variables, you will get an error since Upstart cannot then guarantee uniqueness. For example, if you have a job configuration file foo. Note that to obtain correct restart behaviour, you would need to do something like the following:. Note that " workers. Note further that if any worker fails to start or stop, this wil fail the overall " workers " job.

If you don't want this behaviour, use the " true " trick:. Note that if you have a job which makes use of instance but which may need to be run manually by an administrator, it is possible to "cheat" and allow them to start the job without specifying an explicit instance value:. And this will work even if there is already a running instance of the trickery job assuming the existing instance was started automatically.

This empty value is enough to make that instance unique since there are no other instances with a null instance value! Note that it is possible to specify the signal as a number for example 15 although this should be avoided if at all possible since signal numbers may differ between systems.

Note that if you are running an older version of Upstart without this feature, and you have an application which breaks with the normal conventions for shutdown signal, you can simulate it to some degree by using start-stop-daemon 8 with the --signal option:.

The number of seconds Upstart will wait before killing a process. The default is 5 seconds. If a user job specifies this stanza, it may fail to start should it specify a value greater than the users privilege level allows. For further details on the available limits see init 5 and getrlimit 2.

It is useful for keeping the logic and capability of a job on the system while not having it automatically start at boot-up. Change the jobs scheduling priority from the default. See nice 1. Used to change Upstart's idea of what a "normal" exit status is.

Conventionally, processes exit with status 0 zero to denote success and non-zero to denote failure. If your application can exit with exit status 13 and you want Upstart to consider this as an normal successful exit, then you can specify:. You can even specify signals. Linux has an "Out of Memory" killer facility. This is a feature of the kernel that will detect if a process is consuming increasingly more memory.

Once "triggered", the kernel automatically takes action by killing the rogue process to avoid it impacting the system adversely. Normally the OOM killer regards all processes equally, this stanza advises the kernel to treat this job differently. The "adjustment" value provided to this stanza may be an integer value from very unlikely to be killed by the OOM killer up to very likely to be killed by the OOM killer.

It may also be the special value never to have the job ignored by the OOM killer entirely potentially dangerous unless you really trust the application in all possible system scenarios. Script or process to run after the main process has been spawned, but before the started 7 event has been emitted.

Use this stanza when a delay or some arbitrary condition must be satisfied before an executed job is considered "started". An example is MySQL. After executing it, it may need to perform recovery operations before accepting network traffic. Rather than start dependent services, you can have a post-start like this:. There are times where the cleanup done in pre-start is not enough. Ultimately, the cleanup should be done both pre-start and post-stop , to ensure the service starts with a consistent environment, and does not leave behind anything that it shouldn't.

If it is possible, you'll want to run your daemon with a simple exec line. Something like this:. If you need to do some scripting before starting the daemon, script works fine here. Here is one example of using a script stanza that may be non-obvious:. Because this job is marked respawn , an exit of 0 is "ok" and will not force a respawn only exiting with a non- 0 exit or being killed by an unexpected signal causes a respawn , this script stanza is used to start the optional daemon rpc.

Upstart will not respawn it, but just gracefully see that it has stopped on its own, and return to stopped status. If, however, rpc. Use this stanza to prepare the environment for the job. Another possibility is to cancel the start of the job for some reason. One good reason is that it's clear from the system configuration that a service is not needed:. Note that the " stop " command did not receive any arguments.

This is a shortcut available to jobs where the " stop " command will look at the current environment and determine that you mean to stop the current job. This is safe since the job will not start technically it won't progress beyond the pre-start stage if:.

If this is not the case, just use grep 1 or similar:. See Example of console output for another of example where you can display an error message if the job detects it should not be started.

The pre-stop stanza will be executed before the job's stopping 7 event is emitted and before the main process is killed. However, if the service takes more than kill timeout seconds default, 5 seconds then it will be sent SIGKILL , so if there is anything critical, like a flush to disk, and raising kill timeout is not an option, pre-stop is not a bad place to do it.

You can also use this stanza to cancel the stop, in a similar fashion to the way one can cancel the start in the pre-start. Note that it is possible to specify the signal as a number for example 1 although this should be avoided if at all possible since signal numbers may differ between systems. If you are creating a new Job Configuration File, do not specify the respawn stanza until you are fully satisfied you have specified the expect stanza correctly.

If you do , you will find the behaviour potentially very confusing. This includes running pre-start , post-start and post-stop. Note that pre-stop will not be run. There are a number of reasons why you may or may not want to use this. For most traditional network services this makes good sense. If the tracked process exits for some reason that wasn't the administrator's intent, you probably want to start it back up again. Likewise, for tasks, see below , respawning means that you want that task to be retried until it exits with zero 0 as its exit code.

You may believe that you need to send the service its shutdown command without Upstart being involved, and therefore, you don't want to use respawn because Upstart will keep trying to start your service back up when you told it to shutdown. However, the appropriate way to handle that situation is a pre-stop which runs this shutdown command. Since the job's goal will already be 'stop' when a pre-stop is run, you can shutdown the process through any means, and the process won't be re-spawned even with the respawn stanza.

See stopped 7 for further details. Further note that if the job does not specify the respawn limit stanza as well as the respawn stanza, the job will have the default respawn limit applied see respawn limit. Yes, this is different to a plain respawn : specifying respawn limit does not imply respawn.

Respawning is subject to a limit. To have the job respawn indefinitely, specify an argument of " unlimited ". However, care should be taken using this option: does your service really stop that frequently? Should it? Note that respawn only applies to automatic respawns and not the restart 8 command. Allows the specification of a multi-line block of shell code to be executed. Block is terminated by end script. Note that all processes pre-start , post-stop , et cetera will be run with the group specified.

If this stanza is unspecified, the primary group of the user specified in the setuid block is used. If both stanzas are unspecified, the job will run with its group ID set to 0 in the case of system jobs, and as the primary group of the user in the case of User Jobs. Note that all processes pre-start , post-stop , et cetera will be run as the user specified.

If this stanza is unspecified, the job will run as root in the case of system jobs, and as the user in the case of User Jobs. Note that System jobs using the setuid stanza are still system jobs, and can not be controlled by an unprivileged user, even if the setuid stanza specifies that user. Note that if you specify an invalid username in the setuid stanza, Upstart will log an error if it is in Debug Mode.

For example, if job foo specifies an invalid setuid username:. Although the username is not logged, it is clear there is a problem with the setuid stanza for the specified foo job. This stanza defines the set of Events that will cause the Job to be automatically started. Multiple events are permitted using the operators " and " and " or " and complex expressions may be performed with parentheses within which line breaks are permitted. If you know the order in which the variables are given to the event you may omit the KEY.

VALUE may contain wildcard matches and globs as permitted by fnmatch 3 and may expand the value of any variable defined with the env stanza.

Negation is permitted by using "! Note that if the job is already running and is not an instance job, if the start on condition becomes true again , no further action will be taken. Note that the start on stanza expects a token to follow on the same line.

If no environment variables are specified via KEY to restrict the match, the condition will match all instances of the specified event. See Really understanding start on and stop on for further details. If you are just writing an upstart job that needs to start the service after the basic facilities are up, either of these will work:.

The difference in whether to use the more generic 'runlevel' or the more explicit local-filesystems 7 and net-device-up events should be guided by your job's behaviour.

If your service will come up without a valid network interface for instance, it binds to 0. However if your service requires that a non-loopback interface is configured for some reason i. In addition, services may be aggregated around an abstract job, such as network-services :. The network-services job is a generic job that most network services should follow in releases where it is available. We use the started 7 event so that anything that must be started before all network services can do " start on starting network-services ".

Example: your web app needs memcached to be started before apache :. This stanza defines the set of Events that will cause the Job to be automatically stopped if it is already running. Like the stop on stanza, start on expects a token to follow on the same line :. Or if a generic job is available such as network-services [19]. Note that this also will stop when other-service is restarted, so you will generally want to couple this with the start on condition:.

In concept, a task is just a short lived job. In practice, this is accomplished by changing how the transition from a goal of "stop" to "start" is handled. Without the 'task' keyword, the events that cause the job to start will be unblocked as soon as the job is started.